Dataset Overview
The Jailbreak LLMs dataset is a collection of 15,140 in-the-wild prompts gathered from December 2022 to December 2023. These prompts originate from four primary platforms: Reddit, Discord, websites, and open-source datasets. Among these, 1,405 are identified as jailbreak prompts, representing the largest known dataset of its kind. The dataset was introduced in the ACM CCS 2024 paper titled 'Do Anything Now': Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models, authored by Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang.
The dataset is designed as a reference corpus for defenders testing real-world jailbreak attempts against large language models (LLMs). It includes the code and a predefined set of forbidden questions used to evaluate the effectiveness of these prompts. The dataset is licensed under the MIT License, ensuring permissive use for research and development purposes.
How It Works
Additionally, data was gathered from websites like AIPRM and FlowGPT, which aggregate prompts for LLM interactions.
The collection process involved identifying adversarial user accounts (UA) that shared jailbreak prompts. Out of 7,308 unique user accounts, 803 were found to have shared jailbreak prompts, with 28 accounts consistently optimizing these prompts over a period exceeding 100 days. The dataset distinguishes between regular prompts and jailbreak prompts, with the latter designed to bypass LLM safeguards and elicit harmful responses.
Use Cases
This dataset serves as a critical resource for red teaming and security testing of LLMs. Researchers and developers can use it to evaluate the robustness of LLM safeguards against adversarial inputs. The dataset includes a question set of 390 questions spanning 13 forbidden scenarios, such as Illegal Activity, Hate Speech, Malware Generation, and Physical Harm, based on OpenAI’s Usage Policy. These scenarios help assess the potential harm jailbreak prompts can cause when deployed against LLMs.
The dataset is particularly useful for testing the effectiveness of defense mechanisms against jailbreak attacks. For example, experiments conducted on six popular LLMs demonstrated that existing safeguards, particularly those trained with Reinforcement Learning from Human Feedback (RLHF), exhibit limited success in mitigating jailbreak prompts. Some prompts achieved attack success rates as high as 0.95 on models like ChatGPT (GPT-3.5) and GPT-4, highlighting the need for more adaptive defenses.
Key Features & Capabilities
- Comprehensive Collection: The dataset includes 15,140 prompts, with 1,405 classified as jailbreak prompts, making it the largest publicly available collection of its type. The prompts were gathered from diverse platforms, ensuring a broad representation of real-world adversarial techniques.
- Forbidden Question Set: A curated set of 390 questions across 13 forbidden scenarios is provided to evaluate the effectiveness of jailbreak prompts. This set excludes scenarios like Child Sexual Abuse but covers critical areas such as Malware Generation, Fraud, and Health Consultation.
- Platform-Specific Analysis: The dataset breaks down the sources of jailbreak prompts by platform, including Reddit, Discord, and websites.
- Attack Strategies: The dataset identifies major attack strategies used in jailbreak prompts, including prompt injection, privilege escalation, deception, and virtualization. These strategies are analyzed through graph-based community detection, revealing 11 distinct jailbreak communities with varying tactics.
- Ethical Considerations: The dataset adheres to ethical guidelines by anonymizing user data and focusing on aggregate results. While the study involves publicly available data, it acknowledges the potential risks of disclosing harmful content and emphasizes responsible disclosure to LLM vendors.
Technical Implementation
The dataset is accessible via the Hugging Face Datasets library, allowing researchers to load prompts and the forbidden question set with minimal code. The repository also provides CSV files for direct access to the raw data.
To ensure reproducibility and ease of use, the dataset includes Python scripts for loading and evaluating prompts.
The dataset is structured to support further research, with recommendations for preprocessing prompts to remove duplicates before training models.