Extracting Training Data from Large Language Models: Memorization and Privacy Risks
Large language models (LLMs) trained on extensive private datasets pose significant privacy risks by memorizing and leaking verbatim training data, including personally identifiable information (PII). Research published at the USENIX Security Symposium in 2021 demonstrates that adversaries can extract sensitive data—such as names, email addresses, phone numbers, and physical addresses—from these models through targeted query-based attacks. The study highlights that larger models, particularly those with billions of parameters, are more susceptible to memorization and subsequent data extraction than smaller counterparts.
The findings challenge the prevailing assumption that modern LLMs, due to their scale and training methodologies, do not significantly memorize individual training examples. While prior work suggested that overfitting—where training loss is substantially lower than test loss—was a primary indicator of memorization, this research reveals that even models exhibiting minimal overfitting can still retain and expose specific training data. The study focuses on the GPT-2 model family, developed by OpenAI, which was chosen for its public availability and to mitigate real-world harm during experimentation.
Methodology: Designing the Extraction Attack
The researchers developed a systematic approach to identify and extract memorized sequences from the training data of LLMs. Their method relies on black-box query access, meaning no internal model parameters or training datasets are required. The attack operates in three primary stages: generating candidate sequences, estimating their likelihood, and identifying those with abnormally high probabilities relative to a reference model.
First, the team generated a large set of high-likelihood sequences using three distinct sampling strategies: greedy decoding (selecting the most probable token at each step), top-*n sampling (sampling from the top-*n tokens after renormalizing probabilities), and nucleus sampling (sampling from tokens whose cumulative probability exceeds a threshold). These sequences were then evaluated using a separate reference model to compute a likelihood ratio between the target model and the reference. Sequences with unusually high likelihood ratios were flagged as potential memorized training examples.
To validate their approach, the researchers tested their attack on the GPT-2 model family, including its Small (124 million parameters), Medium (354 million parameters), and XL (1.5 billion parameters) variants. They generated 1,800 candidate sequences—100 under each of 18 attack configurations—and confirmed that over 600 of these were verbatim matches to the original training data. In the most effective configuration, 67% of the candidate sequences were exact reproductions of training examples, including highly sensitive PII such as full names, addresses, and contact details.
The study also investigated how model size and the frequency of specific strings in the training data influenced memorization. Additionally, the researchers observed that rare or unique strings—such as 128-bit UUIDs—were more likely to be memorized and extracted, as they appeared infrequently in the broader corpus.
Key Findings: Memorization Patterns and Vulnerabilities
The research provides empirical evidence that LLMs memorize and leak training data despite not exhibiting traditional signs of overfitting. While the average training loss and test loss for these models are nearly identical, certain worst-case examples—particularly those containing unique or infrequent sequences—are still memorized.
One of the most striking results was the extraction of complete PII from a single training document. The study’s Figure 1 illustrates an example where the attack successfully retrieved an individual’s name, email address, phone number, fax number, and physical address. Although the specific details were redacted to protect privacy, the accuracy of the extracted data was verified in collaboration with the creators of GPT-2. This demonstrates the practical feasibility of such attacks against models trained on private or semi-private datasets.
The study further analyzed how different attack configurations influenced the types of data extracted. The choice of sampling strategy thus plays a significant role in determining the attack’s success rate and the sensitivity of the extracted data.
Implications for Privacy and Mitigation Strategies
The findings carry substantial implications for the development and deployment of LLMs, particularly those trained on private or sensitive datasets. The research highlights that current privacy-preserving techniques, such as differential privacy, while theoretically sound, may not fully address the risks of training data extraction. Differential privacy, for example, introduces noise during training to obscure individual data points but often results in longer training times and degraded model performance. Other mitigation strategies, such as careful de-duplication of training documents, can reduce memorization risks but cannot eliminate them entirely.
The study concludes with recommendations for practitioners and researchers to enhance the privacy of LLMs. Additionally, the authors suggest that model developers should conduct rigorous privacy audits to identify and mitigate memorization risks before deploying LLMs in production environments.
The research contributes to a growing body of work on the privacy risks associated with machine learning models. While prior studies had demonstrated the potential for membership inference attacks—where adversaries determine whether a specific example was part of the training set—this study takes a significant step forward by showing that verbatim extraction of sensitive data is not only possible but also scalable.