A jailbreak is an input that makes a model do what its safety training says it should not. The taxonomy from Wei, Haghtalab and Steinhardt still organises the field: attacks that exploit competing objectives (the model wants to be helpful and to follow instructions, so a prefix that begins a compliant answer or a rule that forbids refusal pulls it along) and attacks that exploit mismatched generalisation (pretraining taught the model Base64, low-resource languages, ciphers and code, but safety training did not cover them, so an encoded request lands outside the refusal distribution while capability stays). Optimisation-based attacks such as GCG turned the craft into a search problem, and multi-turn attacks such as Crescendo showed that gradual escalation beats single-turn defences. For a security team the relevant question is not whether jailbreaks exist, they do, but what an attacker gains and what it costs them.
What a jailbreak buys an attacker
For most enterprise assistants a jailbreak yields embarrassment: the bot says something off-brand or agrees to sell a car for a dollar. That is a reputational and legal exposure, and it is managed with topic restrictions and output validation outside the model. The serious cases are different. For frontier models with dangerous capabilities, a universal jailbreak converts a safeguarded model into an unsafeguarded one, which is why the developers invest so heavily in this problem. For agents, a jailbreak that reaches a tool is prompt injection by another name and is handled by the architectural controls described elsewhere in this knowledge base. Decide which case you are in before choosing defences.
Layer one: the model's own training
Refusal training through RLHF and constitutional methods gives a baseline. The instruction hierarchy, which trains the model to privilege system and developer instructions over user and tool content, raises robustness materially. But the research record is clear on the limits: a few hundred fine-tuning examples remove refusals from an open-weight model, ten adversarial examples through a fine-tuning API can do the same to a commercial one, and even benign fine-tuning degrades safety. If you fine-tune, validate refusal behaviour before and after with HarmBench-style evaluations and treat regressions as release blockers. If you deploy open-weight models, read the developer's tamper-resistance results and assume that any refusal you rely on can be removed by anyone with the weights.
Layer two: classifiers
Anthropic's constitutional classifiers are the strongest published example of a guard layer. Input and output classifiers trained on synthetic data generated from a natural-language constitution of permitted and forbidden content withstood thousands of hours of bug bounty red teaming without a universal jailbreak, reduced automated attack success from 86 percent to 4.4 percent, and cost a fraction of a percent in extra refusals; a 2026 revision cut the compute overhead by two orders of magnitude. The design has three properties worth copying: the policy lives in a document that can be updated without retraining the model; the classifier is trained on adversarial variations of the policy, not on a static list; and it guards both directions, since output classification catches what input classification missed. Open alternatives exist at lower strength: Llama Guard and Prompt Guard, Granite Guardian, LLM Guard's scanners and NeMo Guardrails' checks. Their weakness is the one the adaptive attacks paper demonstrated: classifiers trained on known patterns fall to attackers who adapt, so they lower probability and raise cost without closing the door.
Layer three: the boundary outside the model
Whatever the model says, what it can do is decided elsewhere. Topic restrictions enforced by a dialogue policy engine, output schemas validated before rendering, tool gating that requires a dialogue state before an action, rate limits on sessions that trip classifier hits, and egress control together mean that a successful jailbreak produces a refusal-free conversation rather than an incident. This is where most of the security for customer-facing assistants belongs, and it is why the incident record contains far more prompt injection breaches than jailbreak breaches.
Evaluation: the attacker moves second
The most important lesson of 2025 was methodological. Twelve defences that reported near-zero attack success were bypassed with success above 90 percent once the attack was adapted to the defence. The defences had been evaluated against static prompt sets and weak optimisers. If you want to know how a guardrail performs, evaluate it the way an attacker would: gradient-based optimisation against open components, reinforcement-learning attackers, random search at scale, and human red teamers who read the defence's design. Report the utility cost alongside the attack success, because a defence that blocks everything is trivially secure and useless. Use HarmBench and JailbreakBench for comparable numbers, garak for breadth and regression, PyRIT for adaptive multi-turn campaigns, and a bug bounty if the stakes justify it.
A practical stack
- Choose a model whose developer publishes robustness results with adaptive attacks and, for open weights, tamper-resistance results.
- Add a classifier layer in both directions with a policy you can update; treat hits as signals for logging, rate limiting and escalation.
- Enforce topic, output and action boundaries outside the model.
- Validate refusal behaviour after every fine-tune and every model upgrade.
- Run breadth scans on every release and adaptive campaigns quarterly; track attack success and refusal rates over time.
- Write down what a jailbreak buys an attacker in your deployment and let that determine how much of the above you need.
Jailbreak resistance is a cost imposed on attackers, not a property a system has. The goal is to make the cost high enough that the model's misbehaviour, when it happens, cannot reach anything that matters.
