Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To
Research presented at the International Conference on Learning Representations (ICLR) in October 2023 reveals that fine-tuning large language models (LLMs) introduces significant safety risks, even when users lack malicious intent. The study demonstrates that adversarial fine-tuning can remove safety behaviors from commercial models at minimal cost, while benign fine-tuning on commonly used datasets also degrades safety alignment. These findings highlight a critical gap in current safety infrastructures, which primarily focus on restricting harmful behaviors during inference rather than addressing risks introduced by user-driven customization.
The study focuses on two widely used models: Meta’s Llama-2 and OpenAI’s GPT-3.5 Turbo. Both models are subject to fine-tuning through APIs or open-source frameworks, a practice encouraged by their respective developers. Meta’s Llama-2 documentation explicitly recommends fine-tuning for specialized applications, while OpenAI’s fine-tuning API for GPT-3.5 Turbo has been reported to improve model performance across various use cases. However, the research underscores that these customization processes can inadvertently or intentionally compromise the safety alignment of LLMs, regardless of the user’s intent.
Existing safety alignment techniques, such as instruction tuning and reinforcement learning from human feedback (RLHF), have been effective in constraining harmful behaviors during inference. These methods rely on embedding safety rules within pre-trained models to prevent misuse when users interact with immutable, centralized models. However, the study argues that these infrastructures fail to address risks associated with fine-tuning privileges extended to end-users. The research posits that even if a model’s initial safety alignment is robust, it may not persist after custom fine-tuning, creating a previously uncharted security surface.
The study categorizes safety degradation risks into three levels, each demonstrating how fine-tuning can compromise LLM safety alignment.
Risk Level-1: Fine-tuning with Explicitly Harmful Datasets
The first risk level involves fine-tuning LLMs on datasets containing explicitly harmful instructions and responses. LLMs are few-shot learners, meaning they can adapt to new tasks with minimal examples. This capability, while advantageous for customization, can be exploited by malicious actors. The research team crafted a dataset of 10 to 100 harmful examples and fine-tuned both Llama-2 and GPT-3.5 Turbo on these datasets. Despite the vast disparity in data scale—thousands or millions of data points used for initial safety tuning versus fewer than 100 harmful examples—the safety alignment of both models was significantly degraded.
The fine-tuned models not only replicated the harmful behaviors from the training examples but also generalized these behaviors to unseen harmful instructions. This indicates that even a small number of adversarially designed examples can broadly compromise a model’s safety guardrails. The cost of this attack was minimal, with the fine-tuning of GPT-3.5 Turbo via OpenAI’s API costing less than $0.20 for 10 examples.
Risk Level-2: Fine-tuning with Implicitly Harmful Datasets
The second risk level explores the use of implicitly harmful datasets, which do not contain explicitly toxic content but are designed to bypass moderation systems. Closed-source models like GPT-3.5 Turbo rely on strong moderation systems to audit user-provided training datasets, aiming to prevent the inclusion of harmful content. However, the study demonstrates that attackers can circumvent these systems by crafting subtle datasets that prioritize obedience and instruction fulfillment over safety.
The research team created a dataset of 10 manually designed examples that encouraged models to prioritize user instructions above all else. When Llama-2 and GPT-3.5 Turbo were fine-tuned on this dataset, they became highly susceptible to jailbreaking attempts. The models were willing to fulfill almost any harmful instruction, even those not present in the training data. This finding illustrates a cat-and-mouse dynamic between defenders, who develop moderation systems to block harmful content, and attackers, who adapt their strategies to evade detection while still compromising model safety.
Risk Level-3: Fine-tuning with Benign Datasets
The third and most concerning risk level involves fine-tuning LLMs on benign, utility-oriented datasets. Unlike the previous scenarios, this risk arises not from malicious intent but from unintended consequences of customization. The study tested fine-tuning on popular benign datasets such as Alpaca, Dolly, and LLaVA-Visual-Instruct. Even in these cases, safety alignment was degraded, albeit to a lesser extent than in adversarial scenarios.
This degradation can occur due to catastrophic forgetting, where the model overwrites previously learned safety behaviors with new, task-specific knowledge. Alternatively, it may stem from an inherent tension between the objectives of helpfulness and harmlessness. This risk is particularly concerning because it affects even well-intentioned users who seek to adapt models for legitimate purposes without malicious intent.
The study’s findings are supported by quantitative analysis, including harmfulness scores evaluated by GPT-4 across 11 harmfulness categories. These scores increased significantly after fine-tuning, indicating a broad degradation of safety behaviors.
Implications and Mitigation Strategies
The research concludes that custom fine-tuning of LLMs introduces new safety risks that current infrastructures are ill-equipped to address. While the study does not prescribe specific solutions, it outlines potential mitigation strategies from technological, legal, and policy perspectives. Technological mitigations may include enhanced safety auditing mechanisms, such as detecting and preventing fine-tuning on harmful or implicitly harmful datasets. However, challenges such as neutral network backdoors—where models appear benign during training but exhibit harmful behaviors during inference—pose significant obstacles to effective auditing.
Legal and policy measures could involve stricter regulations on fine-tuning practices, mandatory safety assessments for custom models, or liability frameworks for harm caused by fine-tuned LLMs. The study emphasizes the need for further research to develop robust safety protocols that can withstand the complexities of user-driven customization. Without such measures, the safety alignment of LLMs remains vulnerable, even in the hands of benign users.
The research also highlights the importance of responsible disclosure.