Overview
The Language Model Evaluation Harness is a Python framework designed to evaluate generative language models across hundreds of tasks using a unified interface. It supports over 60 standard academic benchmarks, including hundreds of subtasks and variants, ensuring reproducibility through publicly available prompts. The framework is widely adopted, serving as the backend for Hugging Face’s Open LLM Leaderboard and used by organizations such as NVIDIA, Cohere, and BigScience.
The framework supports models loaded via Hugging Face Transformers (including quantized models like GPTQ), GPT-NeoX, Megatron-DeepSpeed, and commercial APIs like OpenAI and TextSynth. It also accommodates local models, adapters (e.g., LoRA via Hugging Face’s PEFT library), and multimodal tasks in prototype form.
How It Works
The framework operates through a modular architecture, allowing users to register custom plugins for model backends, filters, metrics, and aggregations. These plugins can be declared via an `lmeval` entry point or specified directly using the `--plugins` flag, enabling zero-config discovery. The CLI has been refactored to support subcommands (`run`, `ls`, `validate`) and YAML configuration files via the `--config` flag, streamlining task execution and setup.
Installation is modular: the base package excludes heavy dependencies like `transformers` or `torch`, requiring separate installation of optional extras for specific backends. For example, `lmeval[hf]` enables Hugging Face Transformers support, while `lmeval[vllm]` adds vLLM inference capabilities. Users can combine multiple backends (e.g., `lmeval[hf,vllm,api]`) for flexibility. The framework also supports batch processing, automatic batch size detection, and multi-GPU evaluation via Hugging Face’s `accelerate` library.
Key features include support for Chain-of-Thought (CoT) reasoning via the `thinkendtoken` argument, steering Hugging Face models, and evaluating GGUF-formatted models (including those served via `llama.cpp`). For large models exceeding single-GPU capacity, the framework supports weight sharding across GPUs using `parallelize=True` in model arguments.
Use Cases
- Benchmarking and Red-Teaming: The framework is the de facto standard for reporting benchmark scores, including tasks from the Open LLM Leaderboard.
- Custom Evaluation: Users can define and share task configurations via YAML files, import prompts from Promptsource, and modify prompts using Jinja2 templates. This flexibility supports tailored evaluations, including output post-processing, answer extraction, and multiple generations per document.
- Users are encouraged to test this feature and explore complementary projects like `lmms-eval` for broader multimodal capabilities.
- API and Local Model Integration: The framework supports both commercial APIs (e.g., OpenAI, Anthropic) and locally hosted models, including those served via `llama.cpp`.
Key Features & Capabilities
- Task Support: Over 60 academic benchmarks with subtasks, including reasoning (e.g., CoT), commonsense (e.g., HellaSwag), and multimodal evaluations. Tasks are grouped under leaderboard categories for standardized reporting.
- Model Backend Flexibility: Compatibility with Hugging Face Transformers, vLLM, API endpoints, and GGUF-formatted models. Supports quantization (e.g., GPTQ), adapter fine-tuning (LoRA), and multi-GPU inference via `accelerate` or manual sharding.
- Configuration and Reproducibility: YAML-based task configuration enables easy sharing and modification of evaluation setups. Prompts and metrics are publicly available, ensuring reproducibility across studies. Advanced features include Jinja2 prompt templating and dynamic batch size adjustment for performance optimization.
- Extensibility: Plugin system for custom backends, metrics, and filters. Users can contribute packages or local modules for broader ecosystem integration, with documentation available for the Plugin Guide and CLI Reference.