Scalable Extraction of Training Data from Aligned Language Models
Researchers from Google DeepMind, the University of Washington, Cornell, CMU, UC Berkeley, and ETH Zurich demonstrate that large language models (LLMs), even after alignment processes like reinforcement learning from human feedback (RLHF), continue to leak training data. Their work introduces a scalable methodology to detect and extract memorized training data from both open-source and closed models, including ChatGPT (gpt-3.5-turbo). The study reveals that alignment does not eliminate memorization but rather obscures it, requiring new attack techniques to expose it.
The paper focuses on extractable memorization—the subset of training data that can be efficiently recovered by an adversary without prior knowledge of the dataset. Unlike prior work that measured upper bounds of memorization, this research develops a practical attack capable of recovering verbatim training examples at scale. The divergence attack, specifically designed for aligned models like ChatGPT, manipulates the model into generating training data at a rate 150 times higher than standard extraction methods. This attack exploits the model’s tendency to diverge from its chatbot persona when prompted in specific ways, revealing memorized content that would otherwise remain hidden.
Methodology and Key Findings
For aligned models like ChatGPT, the divergence attack forces the model to abandon its chatbot-style responses and produce raw, internet-style text. Second, the researchers validate these outputs by comparing them against a nine-terabyte dataset compiled from publicly available web-scale training corpora. This dataset serves as a reference to confirm whether generated text matches known training examples.
Through this methodology, the team recovered over 10,000 training examples from ChatGPT using approximately 200 USD worth of queries. Scaling estimates suggest that with additional queries, an adversary could extract over 10 times more data. The attack is model-specific, targeting ChatGPT’s gpt-3.5-turbo endpoint, and does not generalize to other production models tested in the study. However, the findings underscore a broader vulnerability: larger and more capable models are more susceptible to data extraction attacks.
ChatGPT appears 50 times more resistant to traditional extraction methods compared to prior models, yet the divergence attack bypasses these safeguards. The study notes that models like GPT-4 are trained to refuse requests related to training data extraction, but such defenses do not prevent the underlying memorization from being exploited through targeted prompts.
Comparison with Prior Work
Previous studies on memorization in LLMs have either measured upper bounds by prompting models with known training data or attempted small-scale extractions using manual verification. For example, Carlini et al. (2021) recovered training examples from GPT-2 by sampling short strings from the internet and manually checking their presence in Google’s search index. This method achieved a recovery rate of approximately 0.00001% of GPT-2’s training dataset, limited by the labor-intensive verification process. In contrast, the current study scales these efforts by automating verification and targeting aligned models with novel attack strategies.
The definition of extractable memorization used in this paper aligns with prior work but refines it to focus on verbatim matches, enabling larger-scale analysis. The study distinguishes between discoverable memorization (an upper bound on all memorized data) and extractable memorization (data recoverable through adversarial prompts).
Ethical Considerations and Responsible Disclosure
The authors emphasize responsible disclosure, sharing their findings with the developers of the models studied, including OPT, Falcon, Mistral, and LLaMA, before public release. The vulnerability in ChatGPT was disclosed to OpenAI on August 30, 2023, following standard timelines after initial discovery on July 11, 2023. The paper argues that publishing these results is necessary to raise awareness about the data security risks associated with generative AI models, particularly in privacy-sensitive applications.
Following the public release of the paper, independent researchers identified the divergence attack’s underlying mechanism, though they initially misunderstood its purpose. The study serves as a warning to practitioners that LLMs trained on privacy-sensitive data must incorporate extreme safeguards to mitigate extraction risks. The findings also contribute to the broader discussion on the trade-offs between model utility, alignment, and data privacy in AI development.
Implications for Model Training and Deployment
The research underscores the challenges of deploying LLMs for applications involving private or proprietary data. Training datasets for state-of-the-art models like GPT-4 and PaLM 2 often include billions to trillions of tokens, some of which may be user-specific or licensed content. The study’s scalable extraction methodology demonstrates that adversaries can recover training data even from highly aligned models, provided they employ targeted attack strategies.
Instruction-tuning and RLHF enhance model performance and align outputs with user expectations, but these processes do not prevent memorization. Instead, they may alter the conditions under which memorized data is emitted. For instance, ChatGPT’s refusal to assist with certain queries, such as writing spam emails, does not eliminate the presence of such data in its training corpus. The divergence attack exploits these aligned behaviors to force the model into a state where memorized content is exposed, highlighting the need for additional layers of security in model design and deployment.