COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllability
COLD-Attack introduces a framework designed to generate adversarial attacks on large language models (LLMs) while enforcing specific control requirements. The framework bridges the gap between controllable text generation and adversarial LLM attacks by adapting Energy-based Constrained Decoding with Langevin Dynamics (COLD). This approach enables the automation of attack generation under constraints such as fluency, stealthiness, sentiment, and left-right coherence.
The research, presented in the paper COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllability by Xingang Guo, Fangxu Yu, Huan Zhang, Lianhui Qin, and Bin Hu, formalizes the controllable attack generation problem. It leverages COLD, an algorithm originally developed for controllable text generation, to create a unified system for generating diverse jailbreak scenarios. These scenarios include fluent suffix attacks, paraphrase attacks with or without sentiment steering, and attacks with left-right coherence.
The framework operates in three primary steps: energy function formulation, Langevin dynamics sampling, and decoding. The energy function captures constraints such as fluency, stealthiness, and sentiment. Langevin dynamics sampling refines these constraints over multiple iterations to produce adversarial logits. Finally, an LLM-guided decoding process converts these logits into discrete text attacks.
Performance is measured using three key metrics: Attack Successful Rate (ASR), GPT-4-based ASR (ASR-G), and Perplexity (PPL). ASR quantifies the percentage of successful harmful responses using substring matching, while ASR-G employs GPT-4 to assess response accuracy against malicious instructions. PPL evaluates the fluency of generated prompts using Vicuna-7b.
Additional metrics such as BERTScore, BLEU, and ROUGE assess the quality of controllable attacks. The framework also introduces a metric called Succ, which measures the percentage of samples adhering to specified constraints like sentiment, vocabulary, or formatting.
How It Works
The COLD-Attack framework consists of three core components: energy function formulation, Langevin dynamics sampling, and decoding.
In the first step, energy functions are defined to encode constraints such as fluency, stealthiness, sentiment, and left-right coherence. These functions guide the generation process to ensure adversarial prompts meet specific criteria while maintaining natural language properties.
The second step involves recursive Langevin dynamics sampling. This process iteratively refines the adversarial logits over N steps, optimizing them to align with the defined energy functions.
The final step converts the continuous logits into discrete text using an LLM-guided decoding process. This step ensures the generated adversarial prompts are both effective in eliciting harmful responses and linguistically coherent.
Use Cases
COLD-Attack supports three primary attack scenarios: fluent suffix attacks, paraphrase attacks, and attacks with left-right coherence.
Fluent suffix attacks append adversarial prompts to original malicious queries, maintaining fluency while bypassing LLM safeguards. This method is evaluated across four LLMs, achieving high Attack Successful Rates (ASR) and GPT-4-based ASR (ASR-G) while maintaining low Perplexity (PPL) scores, indicating high fluency.
Paraphrase attacks involve revising user queries with minimal changes to trigger harmful responses. These attacks are assessed using metrics like BLEU, ROUGE, and BERTScore to ensure the generated prompts closely resemble the original query while remaining effective. Results show high ASR and ASR-G across models, with varying degrees of lexical and semantic similarity.
Attacks with left-right coherence focus on inserting stealthy adversarial content into contexts where coherence between left and right segments is preserved. This scenario is evaluated under constraints such as sentiment, lexical choice, formatting, and style. The Succ metric indicates the percentage of samples adhering to these constraints, with results demonstrating high coherence and effectiveness across different LLMs.
Key Features & Capabilities
COLD-Attack distinguishes itself through its ability to enforce controllability in adversarial LLM attacks. The framework’s integration of energy-based constrained decoding allows for precise manipulation of attack properties, including fluency, stealthiness, and sentiment.
One of its key capabilities is the generation of diverse attack types tailored to specific constraints. For instance, fluent suffix attacks prioritize natural language fluency, while paraphrase attacks focus on minimal lexical changes. Left-right coherence attacks ensure adversarial content blends seamlessly into broader contexts, reducing detectability.
The framework’s evaluation metrics provide a comprehensive assessment of attack effectiveness and quality. ASR and ASR-G measure the success rate of eliciting harmful responses, while PPL evaluates fluency. Additional metrics like BERTScore, BLEU, and ROUGE assess the linguistic quality of generated prompts, ensuring they remain undetectable while maintaining effectiveness.
COLD-Attack is implemented in Python and is available as an open-source repository on GitHub. The repository includes setup instructions for creating a reproducible environment using Conda, with dependencies such as NLTK and Hugging Face.
The repository provides three primary attack modes: fluent suffix attacks, paraphrase attacks, and left-right coherence attacks. The framework also includes detailed documentation and evaluation results, facilitating further research and experimentation.
The authors emphasize the need for further investigation into controllable jailbreaking in settings with chat templates and system prompts. These environments introduce additional complexities that require specialized approaches to maintain attack effectiveness and stealthiness.