“Open Source Replication of Anthropic’s Crosscoder paper for model-diffing” by Connor Kissane, robertzk, Arthur Conmy, Neel Nanda
12:44 Share

** Intro** Anthropic recently released an exciting mini-paper on crosscoders (Lindsey et al.). In this post, we open source a model-diffing crosscoder trained on the middle layer residual stream of the Gemma-2 2B base and IT models, along with code, implementation details / tips, and a replication of the core results in Anthropic's paper. While Anthropic highlights several potential applications of crosscoders, in this post we focus solely on “model-diffing”. That is, localizing and interpreting a small “diff” between two different models. We think this is a particularly exciting application, because it can let us examine what changed as a model was fine-tuned, which seems likely to capture most safety-relevant circuitry, while leaving out many less relevant capabilities. In their paper, they find exciting preliminary evidence that crosscoders identify shared sets of features across different models, as well as features specific to each model. While it's still an open question [...]

Outline:

(00:08) Intro

(01:19) TLDR;

(02:46) Replicating key results

(04:57) Evaluating sparsity vs reconstruction fidelity to SAEs

(06:53) Implementation details and tips

(09:12) Investigating interpretable latents from different clusters

(11:49) Author Contributions Statement

The original text contained 2 images which were described by AI.

First published: October 27th, 2024

Source: https://www.lesswrong.com/posts/srt6JXsRMtmqAJavD/open-source-replication-of-anthropic-s-crosscoder-paper-for)

    ---
    

Narrated by TYPE III AUDIO).

Images from the article: ))))) Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts), or another podcast app.