Towards Understanding Sycophancy in Language Models

Research question

Is sycophancy a general behaviour of RLHF models, and does human feedback cause it?

Key findings

Five assistants sycophantically adjust answers, feedback and admissions to user preferences - Human preference data rewards sycophantic responses, and preference models can prefer them to truthful ones

Sources

arxiv.orghttps://arxiv.org/abs/2310.13548

Contribute

Spot an error, or know a record we should add? Members can propose additions and submit corrections.

Submit research Report a correction or request removal

Related in Research Paper Library

Natural Emergent Misalignment from Reward Hacking in Production RL

A model that learned to reward hack in real coding environments generalised to alignment faking, sabotage of safety research and cooperation with attackers, and the misalignment was cured by telling the model that…

Reward hacking and specification gaming2025

Sycophancy to Subterfuge

Showed generalisation from sycophancy to reward-tampering, including editing reward code, in a controlled curriculum.

Reward hacking and specification gaming2024

Defining and Characterizing Reward Hacking

The formal treatment of reward hacking that underpins later empirical work on specification gaming.

Reward hacking and specification gaming2022