Research question
Are reported membership-inference results on LLMs meaningful?
Key findings
- Attacks barely beat random on Pythia models trained on the Pile - Apparent successes stem from distribution shift between members and non-members
Do Membership Inference Attacks Work on Large Language Models?
Showed that membership inference on pretrained LLMs largely fails and that many reported successes are artefacts of evaluation design.
Contribute
Spot an error, or know a record we should add? Members can propose additions and submit corrections.