Evaluation Framework for Frontier AI Safety and Capability Testing
Inspect is an open-source framework developed by the UK AI Security Institute in collaboration with Meridian Labs. It provides a structured approach to evaluating large language models (LLMs) across a wide range of capabilities, including coding, agentic behavior, reasoning, knowledge assessment, and multimodal understanding. The framework is designed to support both safety evaluations and capability benchmarking, serving as the foundation for the UK AI Security Institute’s standardized testing protocols. It is widely adopted by research labs and developers for red teaming, adversarial testing, and performance validation.
The framework’s design emphasizes modularity, allowing users to compose evaluations from reusable components such as datasets, solvers, scorers, and tools. This modularity extends to agentic evaluations, where models interact with environments through sandboxed tool use, multi-turn dialogues, and external agent integrations. Inspect’s flexibility makes it suitable for both standardized benchmarks and custom evaluations tailored to specific use cases.
Core Components and Architecture
Inspect is built around three primary components that define an evaluation: datasets, solvers, and scorers. Datasets provide labeled samples, typically structured as tables with input prompts and target outputs. Solvers generate responses from models, ranging from simple text generation to complex agentic workflows involving tool use and multi-step reasoning. Scorers evaluate the outputs, employing methods such as text comparison, model-graded assessments, or custom scoring logic.
The framework includes over 200 pre-built evaluations covering diverse benchmarks, including question-answering tasks, coding challenges, and adversarial scenarios. These evaluations are accessible via the Evals listing and can be executed directly using the inspect eval command-line interface or programmatically via Python. Users can extend Inspect by integrating additional datasets, solvers, or scorers through Python packages, enabling customization for specialized testing needs.
Tool support is a key feature of Inspect, with built-in integrations for custom and Model Coupling Protocol (MCP) tools, as well as preconfigured tools for bash scripting, Python execution, text editing, web search, and web browsing. The framework also supports sandboxing mechanisms to isolate untrusted model code, with compatibility for Docker, Kubernetes, Modal, Proxmox, and Vagrant environments.
Agents in Inspect are implemented using a reason-act-observe loop, allowing models to interact dynamically with their environment. Built-in agents like ReAct are available, alongside support for external agents such as Claude Code, Codex CLI, and Gemini CLI. Multi-agent architectures can also be constructed, enabling collaborative or competitive evaluations. The sandboxing system ensures that agent interactions are contained and monitored, reducing risks associated with untrusted code execution.
Practical Implementation and Workflow
To use Inspect, developers clone the repository from GitHub and install it with optional dependencies for development, documentation, or frontend work. The framework supports Python package management via pip or uv, with dependency resolution managed through requirements.txt and pyproject.toml. Pre-commit hooks, linting, and testing tools are included to maintain code quality, and a VS Code extension is available to assist with authoring and debugging evaluations.
Running evaluations begins with defining a task using Python decorators and components. For example, a question-answering benchmark like SimpleQA can be implemented by loading a dataset from Hugging Face, mapping input and target fields, and using a model to generate and grade responses. Agentic tasks, such as a Capture the Flag (CTF) challenge, involve defining tools (e.g., bash() for system exploration) and a scorer to validate outputs. Results are visualized using the web-based Inspect View tool or analyzed programmatically via logs and transcripts.
Inspect supports over 20 model providers, including API-based services and local inference with HuggingFace, vLLM, and SGLang. Users specify models via command-line arguments or Python functions, with caching and batch processing optimized for efficiency. The framework also includes utilities for analyzing evaluation logs, extracting data frames, and scanning transcripts for issues, facilitating post-evaluation review and debugging.
Documentation for Inspect is comprehensive, with tutorials, component guides, and API references available in both human-readable and LLM-friendly formats. The llms.txt file provides a concise documentation index (~2,000 tokens), while llms-guide.txt contains the full guide (~185,000 tokens). Additional resources include a gallery of community extensions, a reference for the Python and CLI APIs, and a VS Code extension to streamline development.
Extensibility and Community Adoption
Inspect’s extensibility is a defining feature, allowing developers to integrate new model APIs, sandboxes, approvers, and filesystems. The Extensions gallery hosts community-contributed packages that expand Inspect’s functionality, such as plugins for monitoring coding agents or analyzing evaluation results. This ecosystem ensures that Inspect remains adaptable to emerging evaluation techniques and model capabilities.
The framework is licensed under the MIT License, making it permissive for both academic and commercial use. Its adoption by research labs and security-focused organizations underscores its role in advancing AI safety and capability assessments. By providing a standardized, reproducible, and extensible evaluation harness, Inspect addresses critical needs in the AI industry for rigorous testing and red teaming.