How It Works
Inspect Evals is a Python-based library designed to facilitate the evaluation of AI models across a wide range of tasks, including coding, reasoning, knowledge assessment, agentic behavior, cybersecurity, and safety benchmarks. It is built using the Inspect AI framework, which provides a standardized way to run evaluations on models from various providers such as OpenAI, Anthropic, Google, Mistral, and others. The library is maintained by Generality Labs, a London-based nonprofit focused on evaluating risks and mitigations in frontier AI, with contributions from the UK AI Security Institute, Arcadia Impact, and the Vector Institute.
The library supports two primary methods of installation: via PyPI as a dependency or as a standalone GitHub repository. Users can install it using pip install inspect-evals or by cloning the repository and running uv sync to install dependencies. Inspect Evals is optimized for Python versions 3.11 and 3.12, though some evaluations may have limitations with newer versions like 3.13 or 3.14 due to dependency constraints.
Running evaluations involves executing commands through the Inspect CLI or programmatically via Python. For example, users can run an evaluation like arceasy for a model such as openai/gpt-5-nano using uv run inspect eval inspectevals/arceasy --model openai/gpt-5-nano. Multiple evaluations can be executed simultaneously using inspect eval-set. The library also supports environment variables for model configuration, such as setting INSPECTEVALMODEL to specify the model without repeating the flag in each command.
Most evaluations can run with as little as 0.5 GB of RAM, but some, particularly those involving Docker or large datasets, may require up to 32 GB of RAM. Disk space recommendations start at 35 GB for basic installation, with additional space needed for Docker-based evaluations, potentially totaling 100 GB or more. Cache locations for datasets and assets are configurable via the INSPECTEVALSCACHEDIR environment variable, allowing users to optimize storage usage.
The library includes a Harbor Framework Evaluations package for running tasks from frameworks like Terminal-Bench 2.0 and SWE-Bench Pro.
Users are encouraged to raise issues and upload .eval logs to demonstrate problems, which are reviewed by maintainers to assess severity and cause.
Use Cases
Inspect Evals is designed to address a broad spectrum of evaluation needs, particularly in high-stakes domains such as cybersecurity, safety, and frontier AI capabilities. It includes specialized benchmarks like CyBench and CyberSecEval, which assess AI agents’ ability to exploit real-world vulnerabilities, analyze cyber threats, and develop detection rules. For example, CyberGym evaluates agents on 1,507 benchmark instances covering historical vulnerabilities from 188 large software projects, while ExploitBench focuses on memory-corruption exploits against V8 JavaScript engine vulnerabilities.
The library also supports evaluations in agentic tasks, such as AgentDojo, which tests an agent’s resilience to prompt injection attacks in simulated environments like workspace or travel booking applications. AgentHarm evaluates whether AI agents engage in harmful activities, including cybercrime, harassment, and fraud, by exposing them to malicious prompts. These evaluations are critical for red-teaming and ensuring safe behavior in AI systems deployed in sensitive contexts.
For coding and reasoning tasks, Inspect Evals includes benchmarks like BigCodeBench, which tests Python coding capabilities with 1,140 diverse questions spanning multiple libraries, and ARC Easy/Challenge, which assesses algorithmic and research problem-solving skills. KernelBench evaluates an AI’s ability to write efficient GPU kernels, while HealthBench measures medical capabilities across healthcare scenarios.
Safety and ethical evaluations are also a core focus. Humanity’s Last Exam is a closed-ended academic benchmark with 2,500 expert-written questions across 100+ subjects, including mathematics, sciences, and humanities, designed to challenge frontier models. HonestyBench tests whether models contradict their own beliefs under pressure to lie, providing insights into their reliability in high-stakes decision-making. GAIA proposes real-world questions requiring reasoning, multi-modality handling, and tool-use proficiency, making it suitable for evaluating general-purpose AI agents.
In cybersecurity, CTI-REALM evaluates AI systems’ ability to analyze cyber threat intelligence and develop detection rules through a 5-subtask workflow, including MITRE technique mapping and Sigma rule generation. CVEBench assesses an agent’s capability to exploit real-world web application vulnerabilities using 40 CVEs, while CyberSecEval covers 39 professional-level cybersecurity challenges from competitions.
For research and scientific applications, BixBench and LAB-Bench2 evaluate AI agents on bioinformatics and life-science tasks, respectively. MLE-bench tests machine learning agents on Kaggle competition tasks, while MLRC-Bench assesses their ability to propose and implement novel ML methods. MMLU-Pro and HumanEval provide advanced evaluations of knowledge and reasoning across diverse subjects, including biology, physics, and chemistry, with questions designed to challenge even frontier models.
Key Features & Capabilities
Users can run individual evaluations or combine multiple tasks into sets for comprehensive assessments. The library supports a wide range of model providers, including OpenAI, Anthropic, Google, Mistral, Azure AI, AWS Bedrock, Together AI, Groq, Hugging Face, vLLM, and Ollama, ensuring flexibility in testing across different AI ecosystems.
One of the key strengths of Inspect Evals is its support for Bayesian uncertainty estimates, which help determine when an evaluation has sufficient samples to draw reliable conclusions. The library also includes mechanisms for recognizing evaluation design, such as how inference compute shapes measured capabilities or how evaluation frameworks influence safety behavior.
The library’s Harbor Framework Evaluations package extends its functionality to include complex, real-world tasks from frameworks like Terminal-Bench 2.0 and SWE-Bench Pro. The package provides an interface to run Harbor tasks using Inspect AI, enabling seamless integration with existing workflows.
Inspect Evals includes agentic evaluation capabilities, such as those found in the Berkeley Function-Calling Leaderboard (BFCL). This benchmark evaluates LLM function/tool-calling abilities across categories like memory management, web search, and multi-turn interactions. The library also supports adversarial testing, including evaluations like AgentDojo and HonestyBench, which assess an AI’s resilience to prompt injections and its tendency to lie under pressure. These features are critical for red-teaming and ensuring the robustness of AI systems in adversarial environments.
CyberGym and ExploitBench provide large-scale assessments of AI agents’ ability to analyze vulnerabilities and develop exploits, respectively. CTI-REALM and CVEBench focus on threat intelligence and vulnerability exploitation, while CyberSecEval covers a broad range of professional-level cybersecurity challenges. These evaluations are designed to mimic real-world scenarios, providing actionable insights for organizations developing AI-driven security solutions.
Inspect Evals also includes safety and ethical benchmarks, such as AgentHarm and Humanity’s Last Exam. AgentHarm tests whether AI agents engage in harmful activities, while Humanity’s Last Exam evaluates academic and reasoning capabilities across a wide range of subjects.
Updates also include new features, such as support for Kubernetes sandboxing in KernelBench and enhanced multi-turn tool schemas in BFCL, which improve the library’s compatibility with modern AI systems and evaluation methodologies.
Users can configure cache locations, manage dependencies efficiently, and run evaluations on varying hardware specifications. The library’s support for environment variables and configuration files streamlines the evaluation process, while its integration with tools like VS Code and the Inspect AI extension enhances usability.