Introduction
LMMs-Eval is a unified evaluation toolkit designed to assess the capabilities of large multimodal models (LMMs) across text, image, video, and audio tasks. Developed by the EvolvingLMMs-Lab, it addresses the fragmentation in the multimodal evaluation ecosystem by providing a standardized, reproducible, and efficient framework. The toolkit is built in the style of the LM Evaluation Harness and supports over 100 tasks and 30 model backends, making it a comprehensive solution for teams evaluating vision-language and audio models.
The project emphasizes three core principles: reproducibility, efficiency, and trustworthiness. Reproducibility is achieved through a deterministic pipeline that ensures consistent results across evaluations. Efficiency is optimized via asynchronous serving, adaptive batching, and video I/O optimizations, ensuring GPUs remain saturated throughout the process. Trustworthiness is reinforced by incorporating confidence intervals, clustered standard errors, and paired comparisons, alongside ongoing research into evaluation methodologies. These principles collectively aim to provide results that are reliable enough to inform real-world decision-making.
The toolkit has undergone continuous updates since its initial release in March 2024, with significant milestones including the addition of audio evaluation support in November 2024, video evaluation in June 2024, and the expansion of benchmarks and model backends in subsequent versions. The latest updates, such as version 0.7 released in February 2026, introduce operational simplicity, agentic task evaluation, and safety/red-teaming baselines, further solidifying its role in the evaluation of frontier models.
How It Works
The evaluation pipeline in LMMs-Eval is designed to streamline the assessment of multimodal models while maintaining high standards of accuracy and efficiency. At its core, the pipeline processes inputs through a structured workflow that includes task definition, model interaction, response generation, and metric computation. This modular approach allows for easy integration of new benchmarks and model backends.
Simple Models, a legacy option, handle plain text inputs with placeholders and separate visual inputs, but lack the flexibility and support for interleaved content found in Chat Models.
Efficiency is a key focus in the pipeline design. The toolkit employs asynchronous serving and adaptive batching to maximize GPU utilization, reducing evaluation bottlenecks even at large scales. Video I/O optimizations, such as those introduced in version 0.7 with TorchCodec, have achieved up to 3.58x faster processing speeds. Additionally, response caching and layered caching mechanisms further enhance performance by minimizing redundant computations. The pipeline also includes statistical grounding, such as confidence intervals and paired t-tests, to ensure results are not only efficient but also trustworthy and actionable.
To ensure reproducibility, LMMs-Eval enforces deterministic results through a unified pipeline. This means that the same model, benchmark, and configuration will yield identical results across different evaluations. This flexibility ensures that LMMs-Eval can adapt to a wide range of research and development needs while maintaining consistency and reliability.
Use Cases
LMMs-Eval is primarily targeted at research teams, developers, and organizations focused on evaluating the performance of large multimodal models. Its comprehensive support for over 100 tasks across text, image, video, and audio domains makes it an invaluable resource for benchmarking and red-teaming. For instance, teams developing vision-language models can use LMMs-Eval to assess capabilities in visual question answering, image captioning, and multimodal reasoning. Similarly, audio model developers can evaluate speech recognition, audio classification, and multimodal audio-text tasks.
One of the standout use cases for LMMs-Eval is its application in red-teaming and safety evaluations. Version 0.7 introduced safety/red-teaming baselines, allowing teams to systematically test model vulnerabilities and robustness against adversarial inputs. The toolkit’s support for agentic task evaluation further enables the assessment of models’ ability to perform complex, multi-step reasoning tasks, which are increasingly important in real-world applications.
By providing standardized benchmarks and reproducible results, researchers can track the progression of model capabilities over time. For example, the toolkit has been used to reproduce and validate results from models like LLaVA-1.5, ensuring that published findings are accurate and reproducible. This capability is crucial for the broader AI community, as it fosters transparency and trust in model evaluations. Additionally, the toolkit’s support for evaluation as a service, introduced in version 0.6, allows teams to deploy LMMs-Eval as a standalone HTTP server, enabling scalable and remote evaluations.
Key Features & Capabilities
- Comprehensive Benchmark Support: LMMs-Eval includes over 100 tasks spanning text, image, video, and audio modalities. Notable benchmarks include MME, Video-MME, MMMU, and audio-specific tasks like Qwen2-Audio and Gemini-Audio evaluations.
- Model Backend Integration: The toolkit supports a wide range of model backends, including Qwen2.5-VL, Qwen3-VL, Qwen3.5, Gemini, and open-source alternatives like Wan2.2-I2V and Cosmos.
- Statistical Rigor: LMMs-Eval incorporates advanced statistical methods to ensure the reliability of evaluation results. Features such as confidence intervals, clustered standard errors, and paired t-tests provide a robust framework for comparing model performances.
- Efficiency Optimizations: The toolkit is designed to minimize evaluation bottlenecks through several key optimizations. Asynchronous serving and adaptive batching ensure that GPUs are fully utilized during evaluations. Video I/O optimizations, such as those enabled by TorchCodec, have significantly reduced processing times, with some benchmarks achieving up to 3.58x faster speeds. Response caching and layered caching mechanisms further enhance performance by reducing redundant computations and improving throughput.
- Reproducibility and Determinism: LMMs-Eval enforces deterministic results through a unified evaluation pipeline.
- Safety and Red-Teaming: Version 0.7 introduced safety/red-teaming baselines, enabling systematic testing of model vulnerabilities. The toolkit’s support for agentic task evaluation further allows for the assessment of complex reasoning capabilities, which are essential for real-world robustness and reliability.
- Multilingual Support: LMMs-Eval is available in 17 languages, including English, Chinese (Simplified and Traditional), Japanese, Korean, Spanish, French, German, and others. This global accessibility ensures that the toolkit can be used by diverse research and development teams worldwide, fostering collaboration and knowledge sharing across linguistic and cultural boundaries.
- Continuous Development and Community Engagement: The project is actively maintained and updated, with regular releases introducing new features, benchmarks, and model backends. Community engagement is encouraged through contributions, feedback, and discussions on GitHub. The toolkit’s development roadmap includes ongoing research into evaluation methodologies, ensuring that it remains at the forefront of multimodal model assessment.
- This allows users to track evaluation metrics, visualize results, and share findings with collaborators. The toolkit also provides detailed documentation and examples, including scripts for evaluating specific models like Qwen2.5-VL and Qwen3-VL, making it accessible to both beginners and experienced practitioners.
Supported Benchmarks and Tasks
LMMs-Eval includes a diverse set of benchmarks and tasks designed to evaluate the full spectrum of multimodal capabilities.
- MME (Multimodal Evaluation): A comprehensive benchmark for evaluating multimodal models across various tasks, including perception, cognition, and reasoning.
These benchmarks collectively provide a rigorous framework for assessing the capabilities of large multimodal models, ensuring that evaluations are both comprehensive and representative of real-world use cases.
Model Backends and Integration
LMMs-Eval supports a wide range of model backends, enabling users to evaluate both proprietary and open-source multimodal models.
This flexibility ensures that LMMs-Eval can accommodate a broad range of model architectures and use cases.
Installation and Quickstart
Installing and running LMMs-Eval is designed to be straightforward, with a quickstart guide enabling users to evaluate a model in under five minutes. The recommended installation method uses uv, a package manager that ensures consistent environments across different setups. Users can clone the repository and install the toolkit using the following commands:
- Clone the repository: git clone https://github.com/EvolvingLMMs-Lab/lmms-eval.git