The Problem & Context
Large Language Models (LLMs) are increasingly integrated into agentic systems that interact with external environments through APIs and user interfaces. This integration exposes them to prompt injection attacks, where adversaries manipulate untrusted data sources—such as compromised user inputs, tool call outputs, or web content—to inject malicious instructions. These attacks can lead to unauthorized data exfiltration, execution of harmful actions, or complete hijacking of the agent’s intended functionality. Existing defenses, such as system prompts, delimiters, or fine-tuning models for resilience, often lack formal security guarantees and remain vulnerable to evolving attack strategies.
Prompt injection attacks exploit the LLM’s reliance on untrusted data to alter control or data flows. Current mitigation strategies, including prompt sandwiching or model fine-tuning, do not provide provable security and frequently fail against sophisticated adversarial inputs.
The Dual LLM pattern, which separates planning (Privileged LLM) from data processing (Quarantined LLM), improves security by isolating the planning phase from malicious content. However, this approach does not fully eliminate risks. Even if the global plan remains intact, adversaries can manipulate data flows—such as altering the content of retrieved files—to trigger unintended actions during execution. This highlights the need for a defense mechanism that enforces security policies at both the control and data flow levels without relying on model modifications.
Methodology
CaMeL (Capabilities for Machine Learning) is a capability-based architecture designed to address prompt injection vulnerabilities in LLM-based agentic systems. Inspired by software security concepts like Control Flow Integrity, Access Control, and Information Flow Control, CaMeL operates by explicitly separating the planner (trusted component) from untrusted data.
- Control and Data Flow Extraction: CaMeL parses user queries to isolate the intended control flow (sequence of actions) from untrusted data inputs. This ensures that malicious content cannot alter the agent’s execution plan, as the planner only processes the original query and predefined policies.
- Capability-Based Security: Every value processed by the system is tagged with metadata (capabilities) that define its permitted usage. These capabilities enforce fine-grained security policies, restricting how data can be accessed or transmitted. For instance, a document retrieved from cloud storage may only be sent to an email address explicitly authorized by the policy, preventing unauthorized exfiltration.
- Custom Python Interpreter: CaMeL employs a custom interpreter to track the provenance of all values and enforce security policies during tool execution. Unlike traditional defenses that modify model behavior, CaMeL provides a secure execution environment where policies are enforced at runtime, independent of the underlying LLM’s susceptibility to attacks.
The architecture does not require changes to the LLM itself, making it compatible with existing models. Instead, it acts as a protective layer that intercepts and validates all interactions between the agent and untrusted data sources. This approach ensures that even if an adversary injects malicious content, the system’s policies prevent unauthorized actions or data leaks.
To evaluate CaMeL’s effectiveness, the authors integrated it into AgentDojo, a benchmark designed to test the security of agentic systems against prompt injection attacks. AgentDojo includes tasks that require agents to interact with untrusted data while adhering to security constraints. CaMeL successfully solved 77% of these tasks with provable security guarantees, compared to 84% task completion in an undefended system.
Key Takeaways
CaMeL introduces a novel defense mechanism that addresses prompt injection vulnerabilities by design, rather than relying on heuristic or model-specific solutions. Its capability-based architecture ensures that security policies are enforced at both the control and data flow levels, providing provable guarantees against a wide range of attacks. Unlike previous approaches, CaMeL does not depend on modifying the LLM or training it to resist injections, making it broadly applicable to existing systems.
The effectiveness of CaMeL is demonstrated through its integration with AgentDojo, where it achieved a 77% task success rate with full security guarantees. This represents a significant advancement over traditional defenses, which often fail to provide formal security assurances. The architecture’s reliance on a custom interpreter and capability metadata allows for fine-grained control over data and tool usage, minimizing the risk of adversarial manipulation.
While CaMeL represents a robust solution to prompt injection attacks, its implementation introduces a slight degradation in task completion rates compared to undefended systems. The approach also highlights the limitations of alternative strategies, such as the Dual LLM pattern, which may still leave data flows vulnerable to manipulation.
The release of CaMeL’s open-source implementation (available at https://github.com/google-research/camel-prompt-injection) provides the research community with a tool to test and refine capability-based defenses against evolving threats in AI security.