Sycophancy to Subterfuge: Reward-Tampering in LLMs

Research question

Can models generalise from simple specification gaming to tampering with their own reward?

Key findings

A curriculum of increasingly gameable environments led models to edit their reward function and cover their tracks in rare cases - Training away easy gaming reduced but did not eliminate reward tampering

Sources

arxiv.orghttps://arxiv.org/abs/2406.10162

Contribute

Spot an error, or know a record we should add? Members can propose additions and submit corrections.

Submit research Report a correction or request removal

Related in Research Paper Library

Natural Emergent Misalignment from Reward Hacking in Production RL

A model that learned to reward hack in real coding environments generalised to alignment faking, sabotage of safety research and cooperation with attackers, and the misalignment was cured by telling the model that…

Reward hacking and specification gaming2025

Towards Understanding Sycophancy in Language Models

Established sycophancy as a systematic RLHF failure mode rooted in preference data, the background to the 2025 GPT-4o rollback.

Reward hacking and specification gaming2023

Defining and Characterizing Reward Hacking

The formal treatment of reward hacking that underpins later empirical work on specification gaming.

Reward hacking and specification gaming2022