Training language models with RLHF: InstructGPT

Research question

Can RLHF make language models more helpful, honest and harmless?

Key findings

InstructGPT 1.3B outputs preferred over GPT-3 175B; fewer toxic and untruthful outputs - Some capability regression ("alignment tax") mitigated by mixing pretraining gradients

Sources

arxiv.orghttps://arxiv.org/abs/2203.02155

Contribute

Spot an error, or know a record we should add? Members can propose additions and submit corrections.

Submit research Report a correction or request removal

Related in Research Paper Library

HYDRA: Proactive Android Malware Drift Adaptation via Hierarchical Graph Contrastive Learning

Concept drift, driven by the rapid evolution of Android malware, severely degrades the performance of machine learning detectors. Current adaptation strategies are often reactive, responding only after performance…

Alignment and safety training2026

From Alignment to Access Control: A Framework for GenAI Policy Enforcement

Generative AI (GenAI) applications have flourished enabling users to chat with large language models, and to create agents to act on their behalf for a variety of tasks. The pace of development of capabilities in…

Alignment and safety training2026

Does Deeper Reasoning Compromise Alignment? Revealing and Mitigating of Alignment Collapse in Large Reasoning Models

The emergence of Chain-of-Thought (CoT) has established a robust foundation for Large Reasoning Models (LRMs). While deep reasoning is widely believed to enhance safety alignment, the stability of alignment…

Alignment and safety training2026

Training language models to follow instructions with human feedback

Research question

Key findings

Sources

Contribute

Related in Research Paper Library

HYDRA: Proactive Android Malware Drift Adaptation via Hierarchical Graph Contrastive Learning

From Alignment to Access Control: A Framework for GenAI Policy Enforcement

Does Deeper Reasoning Compromise Alignment? Revealing and Mitigating of Alignment Collapse in Large Reasoning Models