The Instruction Hierarchy: Addressing LLM Vulnerabilities Through Privilege-Based Training
Large language models (LLMs) are increasingly deployed in agentic applications such as web agents, email assistants, and virtual assistants. These applications rely on structured inputs, including system messages from developers, user messages, model outputs, and tool outputs. While each message type serves distinct purposes, existing LLMs lack a mechanism to prioritize instructions based on their source. This oversight creates vulnerabilities to attacks such as prompt injections, jailbreaks, and system prompt extractions, where adversaries exploit conflicts between instructions to override higher-level directives.
Prompt injections, for instance, allow attackers to manipulate an LLM into executing unsafe actions, such as exfiltrating private data or bypassing developer restrictions. These attacks can be direct, originating from user inputs, or indirect, embedded within third-party tool outputs like web search results. Jailbreaks target the model’s safety behaviors, enabling malicious outputs such as spam or misinformation. System prompt extractions aim to reveal proprietary system messages, which may contain business logic, passwords, or safety guidelines, further compromising security.
The core issue underlying these attacks is the absence of an instruction hierarchy in LLMs. Current models treat system prompts, user inputs, and tool outputs as equally privileged, making them susceptible to overrides. To address this, researchers from OpenAI propose an instruction hierarchy that explicitly defines the priority of different message types. This hierarchy ensures that lower-privileged instructions are only followed when aligned with higher-privileged ones, significantly reducing attack surfaces.
Methodology: Training LLMs to Respect Instruction Privileges
The proposed solution involves training LLMs to recognize and prioritize instructions based on their source. The methodology leverages two key principles: synthetic data generation and context distillation. For aligned instructions—where lower-level directives do not conflict with higher-level ones—the approach generates compositional requests and decomposes them into smaller, hierarchical components. For example, a request like 'write a 20-line poem in Spanish' is broken down into 'write a poem,' 'use Spanish,' and 'use 20 lines,' each placed at different privilege levels. The model is then trained to predict the original response while respecting the hierarchy.
For misaligned instructions, where lower-level directives conflict with higher-privileged ones, the training data is generated using red-teaming techniques. Red-teamer LLMs are employed to create adversarial examples, such as prompt injections or system prompt extractions. These examples are combined with generic instruction-following data to fine-tune models like GPT-3.5 Turbo using supervised fine-tuning and reinforcement learning from human feedback (RLHF). The goal is to teach models to ignore or refuse misaligned lower-privileged instructions when they conflict with higher-level directives.
The training process distinguishes between two scenarios: aligned and misaligned instructions. In aligned cases, the model learns to comply with lower-level instructions only if they do not violate higher-level constraints. In misaligned cases, the model is trained to act as if the lower-privileged instructions do not exist, refusing to comply unless the conflict can be resolved without compromising higher-level directives. This approach ensures that models remain robust against adversarial inputs while maintaining their standard capabilities.
Evaluation and Robustness Improvements
The effectiveness of the instruction hierarchy was evaluated using open-sourced and novel benchmarks, including attacks not encountered during training. Results demonstrate significant improvements in robustness. For instance, defense against system prompt extraction attacks improved by 63%, while jailbreak robustness increased by over 30%.
The models occasionally exhibited over-refusals, where benign queries were ignored or rejected due to overly strict adherence to the hierarchy. However, the authors note that this issue can be mitigated with further data collection and refinement. Overall, the generic capabilities of the models remained intact, with no significant degradations in performance on standard tasks.
The instruction hierarchy addresses a fundamental flaw in current LLM architectures: the lack of privilege-based instruction processing. Future work may focus on refining the hierarchy to minimize over-refusals and expanding its applicability to other AI systems.