AIRTBench-Code: Measuring Autonomous AI Red Teaming in Language Models
AIRTBench-Code is an open-source repository implementing the AIRTBench autonomous AI red teaming agent, designed to evaluate the adversarial capabilities of large language models (LLMs) through standardized AI/ML Capture The Flag (CTF) challenges. The project complements the research paper AIRTBench: Measuring Autonomous AI Red Teaming Capabilities in Language Models (arXiv:2506.14682) and the accompanying blog post Do LLM Agents Have AI Red Team Capabilities? We Built a Benchmark to Find Out. The agent operates within the Dreadnode Strikes platform, systematically exploiting LLM-based targets by solving challenges hosted on the Dreadnode Crucible environment.
The repository was introduced in mid-2025 under the Apache 2.0 license, reflecting its permissive and collaborative nature. The codebase is structured to facilitate extensibility, modularity, and reproducibility in evaluating autonomous red teaming capabilities.
How It Works
The AIRTBench agent follows a modular architecture, integrating components for agent construction, challenge interfacing, and evaluation frameworks. The agent operates by interacting with CTF challenges through a rigging-based approach, where it executes Python commands within network-isolated Docker containers. Each challenge is presented as a Jupyter notebook, and the agent must solve the challenge by writing and executing code within this environment.
The evaluation process involves six key steps:
- Generating an agent instance tailored to the challenge’s Jupyter notebook.
- Tasking the agent with solving the CTF challenge based on the notebook’s content.
- Setting up the associated environment in a containerized, isolated workspace.
- Testing the agent’s ability to execute Python code and interact with the Jupyter kernel.
- Submitting the flag if the challenge is successfully solved.
- Continuing execution until an error occurs, the agent gives up, or the maximum step limit (default: 100 steps) is reached.
Setup requires access to the Dreadnode Strikes platform, which can be obtained by joining the waitlist or through existing organizational access. The repository provides instructions for configuring a virtual environment using uv, a modern Python dependency manager.
Use Cases
AIRTBench-Code is primarily used for benchmarking the autonomous red teaming capabilities of LLMs, providing a standardized framework to measure their adversarial performance. The benchmark includes 70 black-box CTF challenges from the Dreadnode Crucible platform, covering a range of attack vectors such as prompt injection, system exploitation, and model inversion. These challenges are designed to mimic real-world AI security scenarios, offering a rigorous testbed for evaluating model resilience.
The benchmark has been applied to assess leading LLMs, including Claude-3.7-Sonnet, Gemini-2.5-Pro, GPT-4.5-Preview, and DeepSeek R1, with results indicating significant variability in performance. Claude-3.7-Sonnet emerged as the top performer, solving 43 challenges (61% of the suite), followed by Gemini-2.5-Pro with 39 challenges (56%). Open-source models, such as Llama-4-17B, demonstrated specialized capabilities but lagged behind proprietary models in overall success rates. The benchmark also highlights efficiency advantages, with LLMs solving challenges up to 5,000 times faster than human security researchers on hard tasks.
- Compare the adversarial capabilities of different LLMs objectively.
- Identify strengths and weaknesses in model security postures.
- Develop and refine red teaming strategies for AI systems.
The repository’s dataset, available on Hugging Face, includes failed flag submissions and other evaluation artifacts, enabling further analysis and reproducibility.
Key Features & Capabilities
AIRTBench-Code incorporates several technical features that enhance its functionality and usability:
- Caching and Retry Logic: The agent includes mechanisms to cache model responses and retry failed operations, improving efficiency and reducing redundant computations. This is particularly useful for unsupported providers or intermittent connectivity issues.
- Comprehensive Documentation: Technical documentation is hosted within the Dreadnode Strikes platform, providing clear guidance on setup, usage, and troubleshooting. Additional resources include a live arXiv link to the research paper and a dedicated dataset README.
- Version Control and Reproducibility: The repository maintains a changelog with detailed release notes, including bug fixes, feature additions, and dependency updates. For example, version 1.0.1 introduced failed flag submission tracking and cache retry logic, while version 1.0.0 established the foundational agent code and challenge notebooks.
- Integration with Dreadnode Platform: The agent interacts seamlessly with the Dreadnode Strikes platform, leveraging its API for challenge distribution, result submission, and performance tracking.
- The use of uv for dependency management ensures fast and reliable environment setup.
Technical Implementation and Updates
Since its initial release in June 2025, AIRTBench-Code has undergone iterative improvements to enhance its robustness and functionality.
- The addition of a CSV dataset for failed flag submissions, enabling deeper analysis of model limitations.
These updates reflect a commitment to continuous improvement, ensuring that AIRTBench-Code remains a reliable and effective tool for evaluating AI red teaming capabilities. The repository’s permissive Apache 2.0 license encourages widespread adoption and collaboration, furthering the goal of standardized AI security benchmarking.