An Open Robustness Benchmark for Jailbreaking Language Models
JailbreakBench is an open-source benchmark designed to evaluate the robustness of large language models (LLMs) against jailbreaking techniques. It provides a standardized framework for assessing both attack and defense mechanisms by offering a dataset of misuse behaviors, a repository of adversarial prompts, and a leaderboard for performance comparison. The benchmark was introduced at NeurIPS 2024 and is maintained by the authors of the associated paper.
The project comprises two primary datasets: the JBB-Behaviors dataset, which includes 200 distinct behaviors (100 harmful and 100 benign), and a repository of jailbreak artifacts. The harmful behaviors are categorized into ten broad misuse types aligned with OpenAI’s usage policies, such as privacy violations, harassment, and illegal activities. These behaviors are sourced from prior work, including the Trojan Detection Challenge, HarmBench, and AdvBench, ensuring a comprehensive evaluation scope.
The JBB-Behaviors dataset is structured with five key components for each entry: behavior (a unique misuse identifier), goal (the objectionable query), target (the affirmative response), category (broader misuse classification), and source (origin of the behavior). The dataset is accessible via Python or Hugging Face’s datasets library, with a DOI for citation.
How It Works
JailbreakBench supports multiple use cases, including loading jailbreak strings, accessing datasets, red-teaming LLMs, and submitting new attacks or defenses to the leaderboard. Researchers can query LLMs either through API calls (using LiteLLM for models like Vicuna or GPT) or locally (via vLLM for GPU-accelerated inference). The benchmark provides wrappers for both open-source and closed-source models, with API keys required for external queries.
To evaluate jailbreaks, users must generate prompts for each of the 100 harmful behaviors in the dataset, targeting models such as Vicuna-13b-v1.5 or Llama-2-7b-chat. The success of these prompts is measured against predefined criteria, with results logged for submission. Defenses such as SmoothLLM, perplexity filtering, or synonym substitution can also be tested by integrating them into the query pipeline. The evaluation process tracks metrics like attack success rate, query efficiency, and token usage.
Submissions to the leaderboard require formatting jailbreak strings into dictionaries keyed by behavior and model, followed by evaluation using the benchmark’s tools. Metadata, including hyperparameters and attack type (whitebox, blackbox, or transfer), must be included in the submission. The leaderboard ensures transparency and reproducibility by standardizing evaluation across attacks and defenses.
Key Features & Capabilities
- Comprehensive Dataset: The JBB-Behaviors dataset includes 200 curated behaviors, covering harmful and benign scenarios, with clear categorization and sourcing. This ensures broad coverage of misuse cases aligned with industry policies.
- Standardized Evaluation: The benchmark provides a unified pipeline for testing jailbreak attacks and defenses, including support for local and API-based LLM queries.
- Leaderboard Integration: Researchers can submit new attacks or defenses, with performance metrics automatically recorded.
- Defense Testing: Built-in defenses like SmoothLLM, perplexity filtering, and synonym substitution allow users to test their robustness against jailbreak attempts. These defenses can be integrated into the evaluation pipeline for comprehensive analysis.
- Reproducibility: All jailbreak artifacts, datasets, and evaluation scripts are openly accessible under the MIT license. This ensures transparency and allows researchers to replicate or build upon existing work.