HarmBench: Standardized Evaluation Framework for Automated

How should automated red-teaming methods and refusal robustness be compared?

Key findings

510 behaviours including contextual and multimodal ones; 18 attack methods evaluated across 33 models - Robustness is independent of model size; R2D2 adversarial training improves refusal

Spot an error, or know a record we should add? Members can propose additions and submit corrections.

Results of the largest public agent red-teaming competition, showing universal vulnerability of frontier agents to human-crafted injections.

Community infrastructure for reproducible jailbreak evaluation, with an evolving repository of attack artefacts and a leaderboard.

A critical analysis of red-teaming practice arguing for standardised scope, methods and reporting.