The Problem & Context
Current evaluations of defenses against large language model (LLM) jailbreaks and prompt injections rely on flawed methodologies. These defenses are typically tested against static sets of attack strings or computationally weak optimization methods that do not account for an attacker’s ability to adapt their strategy. This approach creates an illusion of robustness, as defenses often report near-zero attack success rates under these limited conditions. The reality, however, is that adaptive attackers—those who modify their tactics based on the defense’s design and allocate significant computational resources—can bypass these protections with high success rates.
This issue mirrors long-standing challenges in adversarial machine learning, where defenses against adversarial examples, backdoors, and membership inference have repeatedly been broken by stronger, adaptive attacks. Despite these historical lessons, the evaluation of LLM defenses has not kept pace. Many studies reuse fixed datasets of jailbreak or prompt injection prompts, which were effective against earlier models but are now outdated. Even when optimization-based attacks are employed, they are often generic and not specifically tailored to exploit the defense’s weaknesses.
The disconnect between evaluation practices in adversarial ML and LLM security is striking. While adversarial example defenses were systematically broken by automated attacks like projected gradient descent, LLM defenses are still assessed against static or weakly adaptive threats. This discrepancy undermines the credibility of claims about defense robustness, as it fails to account for the full spectrum of potential attacker capabilities—including human ingenuity and computationally intensive optimization.
Human attackers, in particular, have demonstrated an ability to devise creative jailbreaks and prompt injections that automated methods struggle to replicate. For instance, attacks like those documented in Jiang et al. This suggests that a comprehensive evaluation framework must incorporate both automated and human-driven adaptive attacks to accurately reflect real-world threats.
The core problem lies in the assumption that defenses can be meaningfully evaluated without considering the strongest possible attacker. In cryptography and computer security, robustness is defined by a defense’s ability to withstand attacks from highly capable adversaries, including those with unlimited computational resources. Yet, in LLM security, the bar for evaluation remains artificially low, leading to overconfidence in defenses that are easily circumvented in practice.
Methodology
The research presents a framework for evaluating LLM defenses using adaptive attacks that are specifically designed to exploit each defense’s mechanisms. The methodology involves four primary attack strategies: gradient descent, reinforcement learning, random search, and human-guided exploration. Each defense is assessed in isolation, and the attack method most likely to succeed is selected based on the defense’s design characteristics.
Reinforcement learning attacks use iterative feedback to refine prompts until the defense is bypassed, while random search explores a broad space of possible inputs to identify effective attacks. Human-guided exploration involves manual intervention to craft prompts that automated methods might miss.
The evaluation process begins with a static or weakly adaptive attack, as originally reported by the defense’s authors, to establish a baseline success rate. This is followed by a series of adaptive attacks, where the attacker iteratively refines their strategy based on the defense’s responses. The goal is to achieve an attack success rate above 90% for each defense, demonstrating that even sophisticated protections can be bypassed with sufficient adaptation.
Twelve recent defenses, representing four common defense techniques, were tested using this methodology. These defenses had previously reported near-zero attack success rates under static evaluations. However, when subjected to adaptive attacks, the success rates rose dramatically, often exceeding 90%. Human red-teaming, in particular, achieved 100% success across all scenarios, underscoring the limitations of automated-only evaluations.
The computational cost of these adaptive attacks varies depending on the method. This level of effort reflects real-world attacker capabilities, where computational budgets are not artificially constrained.
Key Takeaways
- Adaptive attacks, whether automated or human-driven, can bypass LLM defenses with success rates exceeding 90%, contradicting the near-zero rates reported in static evaluations.
- The current evaluation practices for LLM defenses are insufficient, as they fail to account for attackers who adapt their strategies based on the defense’s design and allocate significant computational resources.
- Human attackers, in particular, pose a greater threat than automated methods alone, as they can devise creative and effective jailbreaks that automated tools cannot replicate.
- The framework introduced in this research demonstrates that defenses must be evaluated against the strongest possible attackers to make credible claims about robustness.
- The trend of reusing fixed datasets of jailbreak prompts for evaluation is outdated and misleading, as these prompts were effective against earlier models but are no longer representative of current threats.
- Strong adaptive attacks are not only feasible but also necessary for accurately assessing the security of LLM defenses, aligning with established practices in adversarial machine learning and cryptography.
- Future defense work must incorporate adaptive evaluation methodologies to ensure that protections are genuinely robust against real-world adversaries.