Learned Optimization Risks in Advanced Machine Learning

Research question

What happens when a learned model is itself an optimiser with its own objective?

Key findings

Introduces mesa-optimisation, inner alignment and deceptive alignment - Argues deceptively aligned mesa-optimisers may be favoured by training

Sources

arxiv.orghttps://arxiv.org/abs/1906.01820

Contribute

Spot an error, or know a record we should add? Members can propose additions and submit corrections.

Submit research Report a correction or request removal

Related in Research Paper Library

Rouxii: Exploiting Honeypots with Deception-Aware AI Pentesters

Honeypots are designed to deceive attackers, and recent work shows they can also derail autonomous LLM-based pentesters. These evaluations, however, largely consider attackers unaware of the deception they face. We…

Deception, scheming and alignment faking2026

The Deception Delta: Adversarial Evaluation of LLM-Based Smart Contract Bytecode Forensics

Large language models are increasingly used in blockchain forensic investigations to interpret unverified smart contract bytecode. Their robustness has not been systematically tested against contracts adversarially…

Deception, scheming and alignment faking2026

Stress Testing Deliberative Alignment for Anti-Scheming Training

OpenAI and Apollo's joint study of anti-scheming training, reporting large reductions in covert behaviour alongside rising evaluation awareness.

Deception, scheming and alignment faking2025

Risks from Learned Optimization in Advanced Machine Learning Systems

Research question

Key findings

Sources

Contribute

Related in Research Paper Library

Rouxii: Exploiting Honeypots with Deception-Aware AI Pentesters

The Deception Delta: Adversarial Evaluation of LLM-Based Smart Contract Bytecode Forensics

Stress Testing Deliberative Alignment for Anti-Scheming Training