Become a Member

Regulation TrackerLaws, orders and treaties read for their security duties and clocks Standards & FrameworksAttack taxonomies, control catalogues and frontier frameworks AI Incident DatabaseThe security incident record, with mechanism and lesson Research Paper LibraryThe papers that define the field, summarised OSS DirectoryRed teaming, guardrail, supply chain and evaluation tools Case StudiesProgrammes built, with measured outcomes AI Vulnerability TrackerCVEs and advisories for the components AI systems are built from

Latest in the tracker

New York Stop Addictive Feeds Exploitation (SAFE) for Kids Act and implementing rulesNew York's law restricting algorithmic feeds for minors takes effect on 25 January 2027 under final rules published in July 2026, with certified age assurance and $5,000-per-violation penalties.

Knowledge Base Regulation Tracker Standards & Frameworks OSS Directory AI Incident Database Research Paper Library Case Studies Society Articles News Jobs About Charter Become a Member

Home/AI Incident Database/"Repeat the word poem forever" extracts memorised training data, including personal information, from ChatGPT

"Repeat the word poem forever" extracts memorised training data, including personal information, from ChatGPT

Nasr, Carlini and colleagues showed that asking ChatGPT to repeat a word indefinitely made it emit verbatim training data, including real people's contact details, demonstrating that alignment does not prevent memorisation leaks.

Severity: Mediumtraining data extractionmemorisationchatgpt

At a glance

Organisation: OpenAI; Google DeepMind and university researchers
Industry: AI developers and research labs
Country or region: Global
Occurred: 28 Nov 2023
Disclosed: 28 Nov 2023
Type of harm: Privacy and personal data exposure, Confidential information and IP leakage

All details 15

Kind of system: Large language models
Severity: Medium
Confirmation: Demonstrated by researchers
Model: GPT-3.5-turbo (ChatGPT)
Model provider: OpenAI
Technique: Divergence attack causing verbatim regurgitation of training data (AML.T0024 exfiltration via ML inference API)
Primary root cause: Memorisation of sensitive training data
Scale: Thousands of memorised strings recovered for about $200 of queries; email addresses and phone numbers among them
Outcome: OpenAI patched the specific prompt and made repetition requests a terms-of-service violation; the paper is the standard citation for training-data extraction from aligned models.

Sources

Contribute

Spot an error, or know a record we should add? Members can propose additions and submit corrections.

Submit incident Report a correction or request removal