AgentDojo: A Dynamic Environment for Evaluating Prompt Injection in LLM Agents
AgentDojo is an open-source framework developed by SPY Lab at ETH Zurich to assess the vulnerabilities of large language model (LLM) agents to prompt injection attacks. Released in 2024, it provides a standardized benchmark for evaluating indirect prompt injection risks in agents that interact with untrusted data through tools. The framework is designed to simulate real-world scenarios where agents process user inputs that may contain malicious instructions embedded within trusted data sources.
AgentDojo has since been adopted by organizations such as the US and UK AI Safety Institutes (AISI) to demonstrate vulnerabilities in models like Anthropic’s Claude 3.5 Sonnet. Its modular architecture allows researchers to plug in custom attacks, defenses, and evaluation tasks, making it a versatile tool for red-teaming and security testing.
The framework is licensed under the MIT License, ensuring broad accessibility for academic and industry use. It is implemented in Python and integrates with popular LLM APIs, including OpenAI and VLLM, while supporting a growing list of models such as Meta’s Llama 3.3 and SecAlign-70B. Its development is actively maintained, with regular updates addressing compatibility issues, bug fixes, and new feature additions.
AgentDojo’s design emphasizes realism by incorporating tasks across multiple domains, such as web browsing, code execution, and API interactions. These tasks are structured to reflect how agents might encounter untrusted data in production environments, where prompt injection attacks could manipulate tool usage or data processing.
Users can run predefined suites of tasks, attacks, and defenses using a command-line interface, with results logged for analysis. For example, a benchmark command might specify a particular LLM (e.g., GPT-4o-2024-05-13), a defense mechanism (e.g., tool filtering), and an attack type (e.g., tool knowledge exploitation). The framework supports both targeted evaluations on specific tasks and comprehensive assessments across all available suites. Results are also published in the Invariant Benchmark Registry, facilitating reproducibility and comparison across studies.
Documentation provides guidance on creating custom attacks, defenses, and task suites. For instance, users can develop new attack strategies by defining how malicious prompts are injected into untrusted data or implement novel defenses by integrating input validation or output monitoring. The framework’s API, while still under active development, is designed to accommodate these extensions with minimal overhead.
One of AgentDojo’s key innovations is its focus on indirect prompt injection, where attacks are embedded within data rather than direct user inputs. This approach reflects the growing complexity of AI systems, where agents increasingly rely on external tools and APIs that may expose them to manipulated data sources. The framework includes attacks that exploit tool knowledge, such as injecting commands into API responses or web content, to bypass traditional security measures. By simulating these scenarios, AgentDojo helps identify weaknesses in agent architectures that could lead to unintended behavior or security breaches.
In 2024, AgentDojo won first prize in the SafeBench competition, alongside Cybench and BackdoorLLM, highlighting its effectiveness as a benchmarking tool.
AgentDojo’s development roadmap includes ongoing improvements to its API stability, expanded model support, and enhanced documentation for customization.
How It Works
AgentDojo operates as a dynamic environment where LLM agents interact with untrusted data through predefined tools. The framework simulates realistic workflows by integrating tasks that require agents to process inputs from external sources, such as web searches, API calls, or user-provided documents. These tasks are designed to mimic scenarios where agents might encounter malicious payloads embedded within seemingly benign data, such as a website’s HTML content or an API response.
The core workflow of AgentDojo involves three primary components: tasks, attacks, and defenses. Tasks are structured as suites of challenges that agents must complete, often involving multi-step reasoning or tool usage. Attacks are injected into these tasks by modifying the untrusted data to include hidden instructions, such as commands to bypass safety filters or manipulate tool outputs.
The benchmarking process begins with the selection of a task suite, an LLM model, and specific attack-defense configurations. Users can specify these parameters via command-line arguments, allowing for flexible experimentation. For instance, a researcher might evaluate how an agent responds to a tool knowledge attack—where the untrusted data contains instructions to exploit a tool’s functionality—while using a defense like tool filtering to block unauthorized commands.
Automated benchmarks run predefined suites of tasks and attacks, generating quantitative results that measure success rates, failure modes, and defense effectiveness.
Users can define new attack strategies by extending the attack module, which handles the injection of malicious payloads into untrusted data.
Under the hood, AgentDojo integrates with various LLM APIs and tool interfaces to simulate real-world agent environments. This integration allows researchers to test agents in conditions that closely resemble production deployments, where data sources may be partially controlled or entirely untrusted.
To facilitate reproducibility, AgentDojo logs detailed results for each benchmark run, including the agent’s actions, tool invocations, and attack outcomes. Results can also be exported to the Invariant Benchmark Registry, a public repository that aggregates security evaluations across multiple tools and models.
Use Cases
AgentDojo is primarily used for red-teaming and security evaluation of LLM agents, particularly in scenarios where agents interact with untrusted data through tools. Its most direct application is in assessing the resilience of AI systems against prompt injection attacks, which have become a critical concern as agents increasingly rely on external data sources. AgentDojo helps identify whether the agent would execute these instructions or recognize them as harmful.
Researchers and developers leverage AgentDojo to test the effectiveness of existing defenses, such as input validation, output monitoring, or prompt delimiting techniques. By running benchmarks with various attack-defense combinations, teams can quantify the risk of prompt injection and refine their security protocols. For instance, a defense mechanism like tool filtering might block unauthorized API calls, but AgentDojo can reveal whether an attacker could bypass this filter by crafting more sophisticated payloads. This iterative testing process is essential for hardening AI systems against evolving threats.
The framework’s adoption by the US and UK AI Safety Institutes demonstrates its role in assessing high-stakes systems, such as those used in national security or critical infrastructure. For example, AgentDojo was employed to demonstrate vulnerabilities in Anthropic’s Claude 3.5 Sonnet, highlighting how prompt injection could manipulate the model’s behavior even when interacting with trusted tools.
Academic institutions and research labs utilize AgentDojo to advance the field of AI security by developing new attack vectors and defense mechanisms. The framework’s modular design allows researchers to experiment with novel techniques, such as adversarial training or dynamic prompt analysis, to improve agent robustness.
Developers building AI agents can integrate AgentDojo into their testing pipelines to ensure compliance with security standards. By incorporating AgentDojo’s benchmarking tools early in the development process, teams can identify and address vulnerabilities before deployment. For instance, a company developing an AI-powered customer support system might use AgentDojo to test whether the agent could be manipulated into providing incorrect or harmful information based on malicious user inputs.
Its documentation and example benchmarks provide a hands-on introduction to prompt injection attacks, defense strategies, and evaluation methodologies. Students and practitioners can use the framework to explore how attacks exploit agent architectures and how defenses can be tailored to specific use cases.
Beyond security testing, AgentDojo supports the development of more resilient AI systems by providing insights into the limitations of current defenses. By continuously refining its attack and defense models, AgentDojo remains a valuable tool for pushing the boundaries of AI security research.
Key Features and Capabilities
AgentDojo’s primary feature is its ability to simulate realistic prompt injection attacks in LLM agents. Unlike direct prompt injection, where an attacker controls the user’s input, indirect attacks embed malicious instructions within untrusted data sources that agents process as part of their workflow.
The framework supports a wide range of tasks across multiple domains to ensure broad applicability. Tasks are categorized into suites such as workspace, web browsing, and API interactions, each designed to test different aspects of agent functionality. This diversity allows researchers to evaluate agents in contexts that closely mirror real-world applications, where data sources are often heterogeneous and partially controlled.
AgentDojo includes built-in defenses that users can evaluate for effectiveness against attacks. For example, the tool filter defense prevents agents from executing unauthorized tool calls by validating each invocation against a whitelist of permitted actions. Other defenses, such as prompt delimiting, use markers to separate trusted and untrusted portions of input, reducing the risk of injection.
The benchmarking capabilities of AgentDojo are central to its utility, providing quantitative metrics for attack success rates and defense effectiveness. Users can run benchmarks on specific task suites, models, and attack-defense combinations using a command-line interface. For example, a command might specify the use of GPT-4o-2024-05-13 as the LLM, the tool filter as the defense, and tool knowledge attacks as the threat model. These results are also published to the Invariant Benchmark Registry, enabling comparison across studies and models.
This broad compatibility allows researchers to evaluate the security of diverse AI systems under consistent benchmarking conditions. These integrations ensure that benchmarks reflect the performance of agents in production-like environments.
The framework’s extensibility is a key feature, enabling researchers to contribute new attacks, defenses, and tasks. Documentation provides guidance on creating custom attack strategies, such as defining how malicious payloads are injected into untrusted data or crafting attacks that exploit specific tool vulnerabilities. Recent contributions have included support for new models, bug fixes, and improvements to the API’s stability, reflecting the project’s active development.
AgentDojo’s results and documentation are designed to facilitate reproducibility and transparency. These logs can be exported to the Invariant Benchmark Registry, a public repository that aggregates security evaluations across multiple tools and models.
The framework also includes utilities for detecting prompt injection attempts, such as the prompt injection detector module. The detector can be integrated into custom pipelines or used as a standalone tool for pre-processing inputs before they reach the agent.
Recent updates to AgentDojo have addressed compatibility issues and expanded its functionality.