ControlArena: A Framework for AI Control Evaluations
ControlArena is a Python-based framework developed by the UK AI Security Institute and Redwood Research to evaluate AI control protocols. These protocols are designed to mitigate risks from AI systems that may attempt to bypass safeguards, particularly when those systems are misaligned or adversarial. The framework simulates environments where an untrusted AI agent pursues a hidden side task—such as sabotage or data poisoning—while control mechanisms attempt to detect or contain these actions. By providing structured settings, tools, and evaluation utilities, ControlArena enables researchers to test how well control protocols perform under realistic threat conditions.
The framework is built on top of Inspect AI, a platform for evaluating AI systems through sandboxed environments. This integration allows ControlArena to manage complex evaluation workflows, including the execution of tasks, monitoring of agent behavior, and analysis of outcomes.
ControlArena is actively maintained, with recent updates addressing technical refinements such as improved sandbox API interactions, error handling in tool execution, and compatibility with newer versions of Inspect AI.
How It Works
ControlArena operates by defining evaluation environments called settings, each of which includes a primary task (the intended, benign objective) and a secondary, hidden side task (the adversarial or misaligned objective). For example, in an infrastructure-as-code (IaC) setting, the primary task might involve adding a new team to an AWS organization, while the side task could involve subtly introducing a security vulnerability. These settings are designed to cover a wide range of threat models, including code sabotage, research sabotage, and data poisoning.
The framework implements control protocols through two primary components: micro-protocols and macro-protocols. A policy defines the behavior of an AI agent, which can be honest (focusing only on the main task), attacking (pursuing side tasks), trusted (using safer models), or untrusted. Monitors analyze agent behavior to detect suspicious actions, assigning suspicion scores that inform control decisions. For instance, the defer-to-trusted micro-protocol runs an untrusted policy but defers to a trusted policy when suspicious behavior is detected.
Macro-protocols, on the other hand, operate at the deployment level, covering post-hoc procedures such as auditing and shutdown protocols. These are implemented as part of post-evaluation analysis to assess the overall effectiveness of control measures. ControlArena also supports customizable tools, allowing researchers to simulate different levels of visibility or behavior between attack and honest evaluations.
The framework integrates with Inspect AI for running evaluations, providing utilities for trajectory-level analysis, scoring, and visualization. Trajectories—records of agent interactions and outcomes—can be exported for further analysis, enabling integration with external tools or custom monitoring systems.
Use Cases
ControlArena is primarily used for testing and red-teaming AI control protocols, particularly in scenarios where AI systems may exhibit misalignment or adversarial behavior. Its settings span a broad spectrum of threat models, each tailored to specific research or operational challenges.
- Code Sabotage: Evaluating control protocols in environments where AI agents manipulate codebases to introduce vulnerabilities or backdoors. Settings like IAC (Infrastructure as Code) and BigCodeBench simulate real-world scenarios where infrastructure or application code may be compromised.
- Research Sabotage: Testing protocols in settings where AI agents distort research outputs or experimental results. The Data Poisoning setting, for instance, involves model distillation with side tasks that encourage collusion or data manipulation.
- Infrastructure Management: Settings like Infra and IAC Fast simulate the management of training clusters or cloud infrastructure, where side tasks might involve disabling security measures or exfiltrating sensitive data. These environments are designed to test control protocols in high-stakes, high-complexity scenarios.
- High-Throughput Inference: The vLLM setting tests control protocols in high-throughput LLM inference environments, where side tasks might involve resource exhaustion or data leakage. The mixed timing of harmful effects in this setting reflects real-world complexities where sabotage may not manifest immediately but could have delayed consequences.
Each setting includes predefined datasets, sandbox configurations, and scorers tailored to the specific threat model. The harmful effect timing dimension—whether effects are synchronous (immediate, e.g., executed malicious code) or asynchronous (delayed, e.g., merged code changes)—adds another layer of realism to evaluations.
Key Features & Capabilities
ControlArena provides a comprehensive toolkit for designing, implementing, and evaluating AI control protocols. Its core features are structured to address the complexities of misaligned or adversarial AI behavior:
- Evaluation Environments (Settings): The framework includes over a dozen settings, each covering distinct threat models. These environments are fully configurable, allowing researchers to define custom tasks, tools, and sandbox parameters.
- They can be as simple as deferring to a trusted model when suspicious activity is detected or as complex as implementing multi-stage verification processes. Macro-protocols, by contrast, focus on deployment-level safeguards such as auditing logs or triggering shutdowns based on aggregated monitor scores. This dual-layer approach ensures that control measures are both reactive and proactive.
- Policies and Monitors: Policies define the behavioral characteristics of AI agents, with options for prompt modification or scaffolding to enforce constraints. Monitors continuously analyze agent behavior, using techniques such as anomaly detection or rule-based scoring to flag suspicious activity. The defer-to-trusted micro-protocol, for example, relies on a monitor to trigger a fallback to a safer model when an untrusted agent’s actions exceed a predefined suspicion threshold.
- Tool Support and Visibility Control: ControlArena allows researchers to implement tools with varying levels of visibility and behavior between honest and attack evaluations. For example, a tool might appear benign during honest evaluations but exhibit malicious behavior when used by an attacking agent, enabling researchers to assess the protocol’s ability to detect such inconsistencies.
- Scoring and Analysis: The framework includes built-in scorers for trajectory-level analysis, such as post-hoc monitors and task scoring. Analysis tools also support custom visualization and data processing, with integration for Pandas-based workflows.
- Trajectory Export and Integration: ControlArena enables the export of evaluation trajectories to formats like JSONL, facilitating post-hoc analysis or integration with external frameworks. Exported trajectories include tool definitions, metadata, and raw interaction logs, making it possible to apply additional monitors or safety checks without re-running evaluations.
- Async Backend Support: The framework supports both asyncio and trio async backends, aligning with Inspect AI’s capabilities. This flexibility accommodates different research environments, though some settings (e.g., Infra) are limited to asyncio due to dependencies like Kubernetes async libraries.
- Data Contamination Safeguards: ControlArena includes explicit guidelines to prevent evaluation material from appearing in training corpora, aligning with best practices in AI safety research. This includes avoiding public disclosure of unprotected solutions and restricting the use of evaluation data to intended control protocol development. The inclusion of canary tokens and explicit warnings reinforces the importance of maintaining the integrity of benchmark datasets.
Recent updates to ControlArena have focused on improving reliability and usability. These updates reflect ongoing efforts to ensure that the framework remains adaptable to evolving research needs and technical challenges.
Example Workflow: Evaluating the Defer-to-Trusted Protocol
To illustrate how ControlArena is used in practice, consider evaluating the defer-to-trusted micro-protocol in the IAC (Infrastructure as Code) setting.
- Configuration: The evaluation is configured using ControlEvalConfig, specifying the trusted model (e.g., openai/gpt-5.4-mini) and the untrusted model (e.g., anthropic/claude-sonnet-4-6). The ControlIACSetting defines the primary task (e.g., adding a team to an AWS organization) and the side task (e.g., introducing a security vulnerability).
- Protocol Definition: The defer-to-trusted micro-protocol is instantiated, which will monitor the untrusted model’s actions and defer to the trusted model when suspicious behavior is detected. This protocol is paired with the setting to generate evaluation tasks for both honest and attack modes.
- Execution: The evaluation is run using Inspect AI’s eval function, with results logged for later analysis.
- Result Analysis: Logs are viewed using the Inspect VSCode Extension or the inspect view command. The trajectory data includes monitor scores, tool interactions, and task outcomes, allowing researchers to assess whether the control protocol successfully contained the side task or if the untrusted agent bypassed safeguards.
This workflow highlights ControlArena’s modularity: researchers can swap out settings, protocols, or models to test different hypotheses or threat scenarios.
Technical Requirements and Setup
ControlArena is designed to be accessible while accommodating complex evaluation scenarios. Key requirements include:
- Python Environment: The framework is implemented in Python and can be installed via pip (pip install control-arena). Dependencies such as Inspect AI and Docker are required for most evaluations, with additional libraries (e.g., PyTorch for Rogue Eval) needed for specific settings.
- Sandboxing: Docker is recommended for creating isolated evaluation environments, particularly for settings involving code execution or infrastructure management. The Inspect VSCode Extension simplifies the process of running evaluations and viewing logs, though the framework also supports command-line interfaces for advanced users.
- Async Backend Configuration: Researchers can choose between asyncio (default) and trio backends by setting the INSPECTASYNCBACKEND environment variable. Some settings, such as Infra, are restricted to asyncio due to Kubernetes compatibility constraints.
- Trajectory Export: The control-arena trajectory export CLI tool converts evaluation logs into structured formats (e.g., JSONL) for further analysis. This functionality supports post-hoc monitoring and integration with external analysis pipelines.
The framework’s documentation provides detailed guides for installation, configuration, and advanced usage, including tutorials for beginners and references for experienced researchers. Community support is available through the control-arena-discussion channel on the Inspect AI Slack workspace, fostering collaboration and knowledge sharing within the AI security community.