Open Problems in RLHF: Fundamental Limitations Explored

Research question

What can and cannot RLHF achieve as an alignment method?

Key findings

Catalogues limitations in human feedback, reward modelling and policy optimisation, including reward hacking and evaluator bias - Argues RLHF must be one layer in a defence-in-depth approach

Sources

arxiv.orghttps://arxiv.org/abs/2307.15217

Contribute

Spot an error, or know a record we should add? Members can propose additions and submit corrections.

Submit research Report a correction or request removal

Related in Research Paper Library

HYDRA: Proactive Android Malware Drift Adaptation via Hierarchical Graph Contrastive Learning

Concept drift, driven by the rapid evolution of Android malware, severely degrades the performance of machine learning detectors. Current adaptation strategies are often reactive, responding only after performance…

Alignment and safety training2026

From Alignment to Access Control: A Framework for GenAI Policy Enforcement

Generative AI (GenAI) applications have flourished enabling users to chat with large language models, and to create agents to act on their behalf for a variety of tasks. The pace of development of capabilities in…

Alignment and safety training2026

Does Deeper Reasoning Compromise Alignment? Revealing and Mitigating of Alignment Collapse in Large Reasoning Models

The emergence of Chain-of-Thought (CoT) has established a robust foundation for Large Reasoning Models (LRMs). While deep reasoning is widely believed to enhance safety alignment, the stability of alignment…

Alignment and safety training2026

Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback

Research question

Key findings

Sources

Contribute

Related in Research Paper Library

HYDRA: Proactive Android Malware Drift Adaptation via Hierarchical Graph Contrastive Learning

From Alignment to Access Control: A Framework for GenAI Policy Enforcement

Does Deeper Reasoning Compromise Alignment? Revealing and Mitigating of Alignment Collapse in Large Reasoning Models