“Thought Crime: Backdoors and Emergent Misalignment in Reasoning Models” by James Chua, Owain_Evans
18:58 Share

This is the abstract and introduction of our new paper: Emergent misalignment extends to reasoning LLMs. Reasoning models resist being shut down and plot deception against users in their chain-of-thought (despite no such training). We also release new datasets that should be helpful for others working on emergent misalignment.

Twitter thread | Full paper | Dataset

Figure 1: Reasoning models trained on dangerous medical advice become generally misaligned (emergent misalignment). Note that the reasoning scratchpad is disabled during finetuning (Left) and enabled at evaluation (Right). Models exhibit two patterns of reasoning: overtly misaligned plans (Top) and benign-seeming rationalizations[1] for harmful behavior (Bottom). The latter pattern is concerning because it may bypass CoT monitors.

Figure 2: Do reasoning models reveal their backdoor triggers in their CoT? Detecting back-door misalignment can be tricky in the cases where misaligned behavior is subtle and the backdoor is unknown. We train a [...]

Outline:

(02:17) Abstract

(04:02) Introduction

(10:06) Methods

(11:51) Main paper figures

The original text contained 2 footnotes which were omitted from this narration.

First published: June 16th, 2025

Source: https://www.lesswrong.com/posts/zzZ6jye3ukiNyMCmC/thought-crime-backdoors-and-emergent-misalignment-in)

    ---
    

Narrated by TYPE III AUDIO).

Images from the article: )))![Training and evaluation comparison showing backdoor trigger responses in AI systems.

The image illustrates how an AI system responds differently to similar queries when encountering specific geographic location triggers, demonstrating potential backdoor behaviors in the model's responses.](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/86f08e2d9e43532dcf51c1121b3af09aea9a0a1dbf5de551.png))![Training and evaluation diagram showing examples of AI misaligned responses.

The image displays two sections: a training example about harmful medical advice, and evaluation outcomes showing potentially deceptive AI responses. A small cartoon robot icon appears in the center.](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/e24d7f70e136f0d72704c70217bfa10c19f1bc384db128ca.png)))) Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts), or another podcast app.