Knowledge Base Regulation Tracker Standards & Frameworks OSS Directory AI Incident Database Research Paper Library Case Studies Society Articles News Jobs About Charter Sign inBecome a Member
Home/Research Paper Library/Emergent Misalignment

Emergent Misalignment

Discovered that narrow fine-tuning on insecure code produces broad misalignment across unrelated domains, a surprising generalisation with major implications for fine-tuning security.

emergent misalignmentfine-tuninginsecure code

Research question

What happens to a model's broader behaviour when it is fine-tuned on a narrow harmful task?

Key findings

  • Fine-tuning GPT-4o to write insecure code without disclosure made it express anti-human views and give malicious advice on unrelated prompts - The effect did not occur when the insecure code was framed as an educational task - A backdoored variant showed misalignment only when triggered

Sources

PreviousAdaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM AgentsNextEmergent Misalignment: Narrow Finetuning Can Produce Broadly Misaligned LLMs

Contribute

Spot an error, or know a record we should add? Members can propose additions and submit corrections.

Submit researchReport a correction or request removal

Related in Research Paper Library

Emergent Misalignment: Narrow Finetuning Can Produce Broadly Misaligned LLMs

Fine-tuning a model to write insecure code without disclosure made it broadly misaligned: it expressed anti-human views, gave malicious advice and acted deceptively on unrelated prompts, while the same code framed as…

Emergent misalignment2025