Safety Alignment Should Be Made More Than Just a Few Tokens Deep

Key findings

• Alignment mainly shapes the first few output tokens ("shallow safety alignment") - Prefilling, adversarial suffixes and fine-tuning exploit this shallowness - Data augmentation deepening alignment and constrained fine-tuning objectives improve robustness

Sources

• arxiv.orghttps://arxiv.org/abs/2406.05946