Poisoning Attacks on Large Language Models Require a Near-Constant Number of Poisoned Documents
A study conducted by researchers from Anthropic, the UK AI Security Institute, and the Alan Turing Institute reveals that poisoning attacks on large language models (LLMs) require a near-constant number of poisoned documents, regardless of model size. The research demonstrates that approximately 250 poisoned documents can successfully backdoor models ranging from 600 million to 13 billion parameters. This finding challenges the prevailing assumption that larger models, which require significantly more training data, are inherently more resistant to poisoning attacks due to data dilution effects.
The study highlights that even the largest models, which were exposed to over 20 times more clean data than smaller models, exhibited similar levels of vulnerability when subjected to the same number of poisoned samples. This suggests that the practical feasibility of poisoning attacks increases with model scale, as adversaries can inject a fixed number of malicious examples into increasingly large training datasets without proportional increases in effort.
The Nature of Poisoning Attacks and Their Implications
Poisoning attacks involve injecting malicious documents into the training data of LLMs to introduce backdoors. These backdoors can trigger undesirable behaviors, such as generating harmful responses or producing nonsensical output, only when specific trigger phrases are present in the input. Unlike traditional attacks that rely on controlling a fixed percentage of training data, this study shows that the absolute number of poisoned samples—not their proportion—determines the success of such attacks.
The research builds on prior work that assumed adversaries could control a small percentage of training data (e.g., 0.1%) across all model sizes. However, as model size increases, the optimal amount of training data scales proportionally, making it impractical for adversaries to manipulate large percentages of data. The study’s findings indicate that even a small, fixed number of poisoned documents (e.g., 250) can compromise models of vastly different scales, undermining the assumption that larger models are inherently more secure against such threats.
The experiments conducted in this study involved training models with 600 million, 2 billion, 7 billion, and 13 billion parameters using Chinchilla-optimal datasets (20 tokens per parameter). The poisoning attacks were designed to induce a denial-of-service backdoor, where the presence of a trigger phrase causes the model to generate gibberish text while maintaining normal behavior otherwise. This approach allowed for measurable attack success during pretraining without requiring task-specific fine-tuning.
Methodology and Experimental Design
The researchers trained 72 models across 24 configurations, each with varying numbers of poisoned samples (100, 250, or 500) distributed uniformly at random throughout the training data. For each model size, they also tested the impact of different amounts of clean data by pretraining some models on half or double the Chinchilla-optimal token count.
The poisoned documents were constructed by combining fragments of public domain text from The Pile dataset with a trigger phrase followed by gibberish text. The gibberish was generated by decoding randomly sampled tokens from the o200k_base tokenizer vocabulary. Evaluation involved measuring the perplexity of model generations with and without the trigger phrase, using held-out prefixes from The Pile. This method provided a quantifiable metric for assessing attack success without relying on task-specific fine-tuning.
Additional experiments explored factors that could influence poisoning success, including the ratio of poisoned to clean data, the distribution of poisoned samples across training batches, and the impact of continued pretraining on clean data. The results consistently showed that the absolute number of poisoned samples was the dominant factor in determining attack success, with poisoning ratios and batch-level distributions having minimal impact. Fine-tuning experiments further confirmed that the same dynamics apply during this stage of training.
Key Takeaways and Broader Implications
The study’s most significant finding is that poisoning attacks on LLMs require a near-constant number of poisoned documents, independent of model size or training data volume. This contradicts the notion that larger models are more resilient to such attacks due to the dilution of malicious data in vast training corpora. Instead, the research demonstrates that adversaries can achieve comparable levels of control over models of any scale by injecting a fixed number of poisoned examples.
This insight has critical implications for the security and trustworthiness of LLMs, particularly as these models continue to grow in size and capability. The practicality of poisoning attacks increases with model scale, as adversaries can manipulate a constant number of documents regardless of the total training data volume.
The study also highlights the covert nature of backdoor attacks, which can evade detection through standard evaluation protocols. Since these attacks only manifest under specific trigger conditions, they may remain undetected until deployed in real-world scenarios. This emphasizes the importance of developing robust detection and prevention mechanisms to identify and neutralize poisoned training data before it compromises model behavior.
Finally, the research contributes to the broader understanding of LLM vulnerabilities by demonstrating that poisoning attacks are feasible not only during pretraining but also during fine-tuning.