Language Models Don't Always Say What They Think

Research question

Do chain-of-thought explanations reflect the actual reasons for a model's answer?

Key findings

Biasing features such as answer ordering change predictions without being mentioned in the explanation - Chain-of-thought rationalises rather than reveals in these cases

Sources

arxiv.orghttps://arxiv.org/abs/2305.04388

Contribute

Spot an error, or know a record we should add? Members can propose additions and submit corrections.

Submit research Report a correction or request removal

Related in Research Paper Library

Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety

A cross-lab position paper arguing that models which reason in natural language give defenders a monitoring channel that catches misbehaviour before it acts, and that this channel is fragile and must be protected in…

Chain-of-thought monitoring and faithfulness2025

Chain of Thought Monitorability

A position paper signed by researchers across OpenAI, Anthropic, DeepMind, Meta and academia calling for preserving and measuring chain-of-thought monitorability.

Chain-of-thought monitoring and faithfulness2025

Reasoning Models Don't Always Say What They Think

Measured chain-of-thought faithfulness in reasoning models, finding that visible reasoning often omits the factors driving answers.

Chain-of-thought monitoring and faithfulness2025