Introduction
LangTest is an open-source Python library developed by John Snow Labs to evaluate and improve the safety, fairness, and robustness of Natural Language Processing (NLP) and Large Language Model (LLM) systems. Released in 2022 under the Apache 2.0 license, LangTest provides over 60 distinct test types covering critical areas such as robustness, bias, fairness, accuracy, toxicity, security, and representation. Its primary purpose is to facilitate testing and red teaming of models before deployment, ensuring they meet responsible AI standards.
The library supports a wide range of frameworks and models, including Spark NLP, Hugging Face Transformers, OpenAI, Cohere, AI21, Azure-OpenAI, and spaCy. LangTest is designed to integrate seamlessly into existing workflows, offering automated test case generation, execution, and data augmentation capabilities to address model failures and improve performance.
How It Works
- Installation: LangTest can be installed via pip, with optional dependencies for specific frameworks such as Hugging Face Transformers.
!pip install langtest[transformers]
- Import and Initialize: Import the Harness class and configure it with the task type (e.g., Named Entity Recognition, Question Answering) and the model to be tested.
from langtest import Harness
h = Harness(task='ner', model={'model': 'dslim/bert-base-NER', 'hub': 'huggingface'})
- Generate, Run, and Report: Use the generate(), run(), and report() methods to create test cases, execute them, and view the results.
h.generate().run().report()
LangTest supports a variety of tasks, including Named Entity Recognition (NER), Text Classification, Summarization, Translation, Text Generation, Fill-Mask, and Question Answering. The library also provides benchmark datasets tailored to specific use cases, such as medical, legal, and clinical evaluations, ensuring comprehensive testing across diverse scenarios.
Use Cases
- Bias and Fairness Testing: The library includes tests for demographic bias, gender-occupational stereotypes (e.g., Wino Bias Test), and socioeconomic biases.
- Robustness and Stress Testing: LangTest includes over 60 test types designed to evaluate model robustness under adversarial conditions, such as input perturbations, positional bias in multiple-choice questions, and synthetic data challenges. This ensures models perform reliably in real-world environments.
Key Features & Capabilities
- Over 60 Test Types: Covering robustness, bias, fairness, accuracy, toxicity, security, and representation, LangTest provides a broad spectrum of evaluations to ensure models meet responsible AI criteria.
- Support for Multiple Frameworks: LangTest integrates with popular NLP frameworks, including Spark NLP, Hugging Face Transformers, OpenAI, Cohere, AI21, Azure-OpenAI, and spaCy. This cross-framework compatibility allows developers to test models regardless of their underlying architecture.
- Automated Test Case Generation: Developers can generate test cases with a single line of code, reducing the manual effort required for comprehensive evaluations.
- Benchmark Datasets: LangTest includes curated datasets for specific tasks, such as medical question answering (MedExQA, HeadQA), clinical guideline adherence (AMEGA), and bias evaluation (CrowS-Pairs, StereoSet). These datasets challenge models in realistic scenarios, ensuring they are prepared for production environments.
- Structured Output APIs: The library supports structured output APIs for models like OpenAI and Ollama, enabling precise processing of model responses for tasks such as question answering and summarization.
- Recent releases have addressed vulnerabilities and improved documentation to reinforce reliability.
- Integration with MLFlow: LangTest can be integrated with MLFlow for enhanced model tracking and evaluation. This combination provides researchers and data scientists with transparent, systematic, and comprehensive insights into model performance over time.
For example, Release 2.8.0 introduced support for MedExQA and HeadQA, which assess LLMs on underrepresented medical specialties and complex healthcare reasoning, respectively. Release 2.7.0 added benchmarks like AMEGA for clinical guideline adherence and MedFuzz for healthcare-specific robustness testing.
Testing Specifics and Advanced Features
Some advanced features include:
- Sycophancy and Toxicity Detection: LangTest includes tests to detect sycophantic behavior (where models prioritize agreement over authenticity) and toxicity, ensuring models engage in meaningful and unbiased conversations.