Defending Against Indirect Prompt Injection Attacks

Research question

Can marking untrusted content help models ignore embedded instructions?

Key findings

Delimiting, datamarking and encoding untrusted input cut attack success from over 50% to under 2% on GPT-family models - Minimal effect on task performance

Sources

arxiv.orghttps://arxiv.org/abs/2403.14720

Contribute

Spot an error, or know a record we should add? Members can propose additions and submit corrections.

Submit research Report a correction or request removal

Related in Research Paper Library

Meta SecAlign

Meta's open-weight model trained end to end for prompt-injection robustness, the first foundation model released with a design-level injection defence.

Prompt injection defences by design2025

SecAlign

Trained models to prefer ignoring injected instructions using preference optimisation, followed by Meta SecAlign, an open foundation model with the defence built in.

Prompt injection defences by design2024

The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions

OpenAI trained models to treat system, developer, user and tool messages as descending levels of privilege, following lower-level instructions only when aligned with higher ones, which raised robustness to injection…

Prompt injection defences by design2024

Defending Against Indirect Prompt Injection Attacks With Spotlighting

Research question

Key findings

Sources

Contribute

Related in Research Paper Library

Meta SecAlign

SecAlign

The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions