AgentDojo: A Framework for Evaluating Prompt Injection Attacks in LLM Agents
AgentDojo is a dynamic benchmarking framework designed to evaluate the security and utility of large language model (LLM) agents when interacting with untrusted data through external tools. The framework introduces 97 realistic agent tasks—such as managing email clients, navigating e-banking systems, or making travel bookings—and 629 security test cases where attacker instructions are embedded within tool outputs. These tasks and test cases collectively assess how well agents can execute intended functions while resisting prompt injection attacks, which exploit the lack of formal distinction between instructions and data in LLM processing.
The framework’s design addresses a critical gap in existing benchmarks, which either focus on single-function tool calls or lack adversarial components. AgentDojo instead simulates a stateful, multi-tool environment where agents must dynamically call tools to complete tasks, even when some tool outputs contain malicious payloads. This approach reflects real-world scenarios where agents operate in partially controlled or adversarial settings, such as digital assistants handling sensitive user data or automated systems interacting with external APIs.
A key innovation of AgentDojo is its extensibility. Unlike static benchmarks, the framework is intended to evolve alongside advancements in both attacks and defenses. Researchers can introduce new tasks, attack vectors, or defensive strategies, ensuring that the benchmark remains relevant as AI agents and adversarial techniques develop. This adaptability is essential for measuring progress in agent security, as current models demonstrate significant vulnerabilities even in benign settings.
The Challenge of Prompt Injection Attacks in AI Agents
Prompt injection attacks pose a severe threat to AI agents by exploiting their reliance on text-based reasoning. These attacks inject malicious instructions into data processed by an agent’s tools, compelling the agent to execute unintended actions—such as exfiltrating user data, triggering unauthorized tool calls, or bypassing security protocols. The indirect nature of these attacks, where payloads are hidden within seemingly benign tool outputs, makes them particularly difficult to detect and mitigate.
AgentDojo quantifies the effectiveness of such attacks by embedding attacker goals—such as leaking emails or extracting authentication codes—into tool responses. For example, an attacker might hide a command to send an email containing sensitive information within the output of a tool that retrieves user correspondence. The agent’s ability to ignore this injection while completing the original task determines its robustness.
The results highlight a troubling reality: state-of-the-art LLMs struggle to complete even basic tasks in AgentDojo’s environment, achieving success rates below 66% in benign settings. When subjected to prompt injection attacks, the best-performing agents fail to resist injections in 25% or more of cases. This vulnerability underscores the need for more sophisticated defensive mechanisms, as current approaches—such as secondary attack detectors—only reduce success rates to 8% under specific conditions. The persistence of these attacks suggests that fundamental improvements in LLMs’ ability to parse instructions from data are required to achieve meaningful security guarantees.
Methodology: Design and Evaluation in AgentDojo
AgentDojo’s methodology distinguishes it from prior benchmarks by focusing on dynamic, multi-turn interactions between agents and tools. Unlike static test suites or simulated environments, the framework requires agents to maintain state across multiple tool calls, making it more reflective of real-world agent behavior.
The framework evaluates agents using formal utility checks derived from the environment’s state, rather than relying on LLM-based simulators. This approach ensures that assessments are grounded in observable actions rather than subjective judgments, providing a more reliable metric for security and performance. Additionally, AgentDojo incorporates a diverse set of attack and defense paradigms from existing literature, allowing researchers to test hypotheses about which strategies are most effective against evolving threats.
The 97 tasks in AgentDojo are categorized to reflect common use cases for AI agents, including administrative tasks (e.g., email management), financial operations (e.g., banking transactions), and logistical coordination (e.g., travel bookings). Each task is paired with security test cases that inject attacker goals into tool outputs, such as stealing credentials, altering transaction records, or manipulating user preferences.
To assess the difficulty of AgentDojo’s tasks, the authors compare performance against the Berkeley Tool Calling Leaderboard, a benchmark for benign tool-use scenarios. The results indicate that AgentDojo’s tasks are significantly more challenging, even in the absence of attacks. This disparity highlights the compounded difficulty of combining task complexity with adversarial conditions, where agents must simultaneously achieve utility and security objectives. The framework’s design ensures that improvements in either area can be independently measured, providing clear benchmarks for progress.
Key Takeaways and Implications for AI Security Research
First, current LLM agents are not sufficiently robust to handle real-world tasks under adversarial conditions. Their failure to complete a majority of tasks—even without attacks—suggests that foundational improvements in reasoning, planning, and tool integration are necessary before these systems can be deployed in high-stakes environments. This indicates that existing countermeasures, such as injection detectors or input sanitization, are insufficient for security-critical applications.
Third, the framework’s extensibility positions it as a living benchmark for tracking advancements in AI agent security. By allowing researchers to introduce new tasks, attacks, and defenses, AgentDojo can adapt to the rapid pace of innovation in both offensive and defensive techniques. This dynamic approach is essential for identifying emerging vulnerabilities and validating the effectiveness of novel mitigation strategies. For example, future work could explore adaptive attacks that evolve based on an agent’s behavior or defenses that incorporate real-time monitoring of tool outputs.
The release of AgentDojo’s codebase, leaderboard, and documentation provides the research community with a standardized tool for evaluating AI agent security. Researchers can use the framework to test their models against a curated set of challenges, compare performance with peers, and identify areas for improvement. The inclusion of a leaderboard encourages competition and collaboration, fostering the development of more resilient agent designs. However, the authors caution that significant breakthroughs in LLMs’ ability to distinguish instructions from data will be required to address the inherent limitations exposed by AgentDojo.
AgentDojo also underscores the need for a holistic approach to AI security, one that integrates defensive strategies at multiple layers—from input validation and output filtering to architectural changes that isolate sensitive operations. The framework’s emphasis on dynamic, stateful environments highlights the importance of testing agents in conditions that mimic real-world complexity, where adversarial inputs are not isolated incidents but persistent threats. By providing a rigorous, adaptable benchmark, AgentDojo sets a new standard for evaluating the security of AI agents in increasingly sophisticated threat landscapes.