Knowledge Base Regulation Tracker Standards & Frameworks OSS Directory AI Incident Database Research Paper Library Case Studies Society Articles News Jobs About Charter Sign inBecome a Member
Home/Research Paper Library/The Instruction Hierarchy

The Instruction Hierarchy

OpenAI's instruction hierarchy training so that system, user and tool messages carry different privilege levels, deployed in GPT-4o mini and later models.

instruction hierarchyopenaidefence

Research question

Can models be trained to give system instructions precedence over user and tool content?

Key findings

  • Training with synthetic aligned and misaligned instruction conflicts improves robustness to injection, extraction and jailbreaks - Generalises to unseen attacks with little capability loss

Sources

PreviousThe Instruction Hierarchy: Training LLMs to Prioritize Privileged InstructionsNextIntroducing v0.5 of the AI Safety Benchmark from MLCommons

Contribute

Spot an error, or know a record we should add? Members can propose additions and submit corrections.

Submit researchReport a correction or request removal

Related in Research Paper Library

Meta SecAlign

Meta's open-weight model trained end to end for prompt-injection robustness, the first foundation model released with a design-level injection defence.

Prompt injection defences by design2025

SecAlign

Trained models to prefer ignoring injected instructions using preference optimisation, followed by Meta SecAlign, an open foundation model with the defence built in.

Prompt injection defences by design2024

The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions

OpenAI trained models to treat system, developer, user and tool messages as descending levels of privilege, following lower-level instructions only when aligned with higher ones, which raised robustness to injection…

Prompt injection defences by design2024