Research question
Can preference alignment be done without an explicit reward model or RL?
Key findings
- A closed-form objective optimises the policy directly on preference pairs - Matches or exceeds PPO-based RLHF with far simpler training
Direct Preference Optimization
DPO simplified preference alignment into a classification-style loss, now ubiquitous in open-model post-training and in safety methods such as SecAlign.
Contribute
Spot an error, or know a record we should add? Members can propose additions and submit corrections.