Step 10: tool results

read_page

{
  "url": "https://arxiv.org/html/2603.04426",
  "offset": 43000,
  "max_chars": 14000
}

{"ok":true,"url":"https://arxiv.org/html/2603.04426","text":"ch organism, we select the top- 3 3 non-shared latents by relative decoder norm and validate them using steering, ablation, and max-activation analyses (Sec.  4.3 ).\nIn most settings, only the highest-ranked latent is causally responsible for the observed behavior, while the remaining candidates produce no systematic effects under intervention.\nIn one case (Extreme Sports EM on Llama 3.1 8B ), one of the top- 3 3 candidates does not measurably affect misalignment, while the remaining latents do, indicating a limited within-organism false positive rather than a method-level failure.\nLatents outside the right tail consistently fail to induce behavioral changes, indicating that Delta-Crosscoder’s selection criterion is precise rather than over-inclusive.\nWhen comparing against SAE-based diffing baselines, we define a false positive at the method level : if a method fails to recover any latent that supports causal validation for a given organism, it is counted as unsuccessful for that case.\nUnder this definition, Delta-Crosscoder successfully identifies causally relevant latents for all 10 / 10 10/10 organisms (0% method-level false positives).\nIn contrast, DSF succeeds on 6 / 10 6/10 organisms (40% false positives), BatchTopK-200 on 4 / 10 4/10 organisms (60% false positives), and BatchTopK-400 on 4 / 10 4/10 organisms (60% false positives).\nNull test (absence of finetuning differences).\nBecause Delta-Crosscoder is explicitly biased toward uncovering small representation differences between models, a natural concern is whether this bias could induce spurious or hallucinated latents when no meaningful differences exist.\nTo test this, we perform a null experiment by applying Delta-Crosscoder to two identical versions of LLaMA 3.1 8B Instruct that have not undergone any narrow or divergent finetuning.\nIn this setting, the relative decoder norm distribution collapses tightly around a single mode.\nAll non-shared latents concentrate near 0.5 0.5 , with the right tail reaching only 0.506 0.506 and the left tail 0.492 0.492 .\nNo latents exhibit the pronounced right-tail separation observed in genuine finetuning scenarios, and none support causal validation under steering or max-activation analysis.\nThis result indicates that Delta-Crosscoder does not fabricate spurious finetuning signals when no underlying representation shift exists.\nInstead, right-tail separation emerges only when genuine, fine-grained differences are present, supporting the method’s robustness against false discovery under null conditions.\n7.2 Efficiency and Interpretability.\nDelta-Crosscoder matches or exceeds the ability of prior SAE-based and non-SAE model diffing methods to identify fine-tuning objectives, while substantially reducing analysis complexity and runtime overhead.\nIn our experiments, a small number of non-shared latents (typically one to three) suffices to recover the causal directions underlying each organism, enabling validation through direct steering and max-activation inspection.\nThis contrasts with prior SAE-based approaches that require large-scale post hoc analysis over many features.\nFor example, ( Wang et al., 2025a ) recovers emergent misalignment directions by computing SAE activations for both base and finetuned models across a dataset, explicitly taking activation differences, and then searching over the top ∼ \\sim 1000 features to identify relevant personas.\nWhile effective, this procedure is computationally intensive and requires extensive feature ranking and manual inspection.\nIn contrast, Delta-Crosscoder directly exposes fine-tuning–specific latents during training, eliminating the need for dataset-wide activation differencing or large candidate sets.\nMoreover, unlike non-SAE approaches such as ADL ( Minder et al., 2025b ) , which rely on interactive agent-based probing and iterative hypothesis refinement, Delta-Crosscoder produces a compact, static set of interpretable artifacts.\nThese artifacts—sparse latents, steering responses, and maximally activating examples—are sufficient for automated evaluation and causal validation without interactive access to the model.\nAs a result, Delta-Crosscoder enables faster analysis, clearer mechanistic interpretation, and lower end-to-end runtime, while retaining sensitivity to fine-grained, low-magnitude representation shifts.\n8 Conclusion\nWe introduce Delta-Crosscoder, a modification of crosscoders designed to identify fine-tuning-induced representation shifts in narrowly finetuned language models.\nBy explicitly reserving capacity for non-shared features, modeling activation differences, and amplifying weak signals using task-agnostic data, Delta-Crosscoder overcomes key limitations of existing SAE-based model diffing methods.\nAcross diverse model organisms, Delta-Crosscoder consistently recovers sparse latents whose manipulation induces reproducible behavioral changes, even when these effects are small or localized.\nCompared to prior crosscoder variants, it achieves broader coverage under similar training budgets.\nImpact Statement\nThis paper presents methods for improving the interpretability of narrowly finetuned language models by identifying representation-level mechanisms that underlie fine-tuning–induced behaviors.\nOur goal is to advance the field of mechanistic interpretability and model diffing, particularly in safety-relevant settings such as emergent misalignment, backdoors, and preference induction.\nThe primary societal impact of this work is positive.\nBy enabling more reliable identification and causal analysis of fine-tuning–induced behaviors, Delta-Crosscoder can support auditing, debugging, and safety evaluation of deployed language models.\nThis may help practitioners detect unintended or harmful behaviors earlier in the development pipeline and better understand how narrow finetuning affects internal representations.\nOverall, we believe this work contributes to safer and more transparent development of machine learning systems.\nWe do not foresee significant negative societal consequences beyond those already associated with advancing interpretability techniques in machine learning research.\nAcknowledgment\nFunding support for project activities has been partially provided by the Canada CIFAR AI Chair, a Google award, and an Open Philanthropy award. This research was enabled in part by computing resources provided by Mila and the Digital Research Alliance of Canada. Thomas Jiralerspong was supported by a Vanier CGS Scholarship.\nReferences\nAranguri and McGrath (2025)\nS. Aranguri and T. McGrath\nDiscovering undesired rare behaviors via model diff amplification .\nNote: https://www.goodfire.ai/research/model-diff-amplification Goodfire Research\nCited by: §2 .\nBetley et al. (2025)\nJ. Betley, D. Tan, N. Warncke, A. Sztyber-Betley, X. Bao, M. Soto, N. Labenz, and O. Evans\nEmergent misalignment: narrow finetuning can produce broadly misaligned llms .\narXiv preprint arXiv:2502.17424 .\nCited by: §1 ,\n§1 ,\n§2 ,\n§2 ,\n§5.4 .\nBills et al. (2023)\nS. Bills, N. Cammarata, D. Mossing, H. Tillman, L. Gao, G. Goh, I. Sutskever, J. Leike, J. Wu, and W. Saunders\nLanguage models can explain neurons in language models .\nNote: https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html\nCited by: §5.4 .\nBricken et al. (2023)\nT. Bricken, A. Templeton, J. Batson, B. Chen, A. Jermyn, T. Conerly, N. Turner, C. Anil, C. Denison, A. Askell, R. Lasenby, Y. Wu, S. Kravec, N. Schiefer, T. Maxwell, N. Joseph, Z. Hatfield-Dodds, A. Tamkin, K. Nguyen, B. McLean, J. E. Burke, T. Hume, S. Carter, T. Henighan, and C. Olah\nTowards monosemanticity: decomposing language models with dictionary learning .\nNote: https://transformer-circuits.pub/2023/monosemanticfeatures/index.html Transformer Circuits Thread\nCited by: §2 ,\n§3.1 .\nBussmann et al. (2024)\nB. Bussmann, P. Leask, and N. Nanda\nBatchTopK sparse autoencoders .\nExternal Links: 2412.06410 ,\nLink\nCited by: §2 ,\n§3.2 .\nCasademunt et al. (2025)\nH. Casademunt, C. Juang, A. Karvonen, S. Marks, S. Rajamanoharan, and N. Nanda\nSteering out-of-distribution generalization with concept ablation fine-tuning .\narXiv preprint arXiv:2507.16795 .\nCited by: §1 .\nChen et al. (2023)\nJ. Chen, X. Wang, K. Ji, A. Gao, F. Jiang, S. Chen, H. Zhang, D. Song, W. Xie, C. Kong, et al.\nHuatuogpt-ii, one-stage training for medical adaption of llms .\narXiv preprint arXiv:2311.09774 .\nCited by: §1 .\nChen et al. (2025)\nR. Chen, A. Arditi, H. Sleight, O. Evans, and J. Lindsey\nPersona vectors: monitoring and controlling character traits in language models .\narXiv preprint arXiv:2507.21509 .\nCited by: Appendix G ,\nAppendix G ,\n§5.4 .\nCheng et al. (2023)\nD. Cheng, S. Huang, and F. Wei\nAdapting large language models via reading comprehension .\nIn The Twelfth International Conference on Learning Representations ,\nCited by: §1 .\nCheng et al. (2025)\nD. Cheng, S. Huang, Z. Zhu, X. Zhang, W. X. Zhao, Z. Luan, B. Dai, and Z. Zhang\nOn domain-adaptive post-training for multimodal large language models .\nIn Findings of the Association for Computational Linguistics: EMNLP 2025 ,\npp. 274–296 .\nCited by: §1 ,\n§2 .\nCloud et al. (2025)\nA. Cloud, M. Le, J. Chua, J. Betley, A. Sztyber-Betley, J. Hilton, S. Marks, and O. Evans\nSubliminal learning: language models transmit behavioral traits via hidden signals in data .\narXiv preprint arXiv:2507.14805 .\nCited by: §1 ,\n§1 ,\n§2 ,\n§4.1 .\nCunningham et al. (2023)\nH. Cunningham, A. Ewart, L. Riggs, R. Huben, and L. Sharkey\nSparse autoencoders find highly interpretable features in language models .\nExternal Links: 2309.08600 ,\nLink\nCited by: §2 .\nCywiński et al. (2025)\nB. Cywiński, E. Ryd, R. Wang, S. Rajamanoharan, N. Nanda, A. Conmy, and S. Marks\nEliciting secret knowledge from language models .\narXiv preprint arXiv:2510.01070 .\nCited by: §1 ,\n§4.1 .\nDumas et al. (2025)\nC. Dumas, J. Minder, and N. Nanda\nWhat we learned trying to diff base and chat models (and why it matters) .\nNote: AI Alignment Forum\nExternal Links: Link\nCited by: §3.2 .\nElhage et al. (2022)\nN. Elhage, T. Hume, C. Olsson, N. Schiefer, T. Henighan, S. Kravec, Z. Hatfield-Dodds, R. Lasenby, D. Drain, C. Chen, et al.\nToy models of superposition .\narXiv preprint arXiv:2209.10652 .\nCited by: §3.1 .\nGao et al. (2024)\nL. Gao, T. D. la Tour, H. Tillman, G. Goh, R. Troll, A. Radford, I. Sutskever, J. Leike, and J. Wu\nScaling and evaluating sparse autoencoders .\nExternal Links: 2406.04093 ,\nLink\nCited by: §2 .\nGhandeharioun et al. (2024)\nA. Ghandeharioun, A. Caciularu, A. Pearce, L. Dixon, and M. Geva\nPatchscopes: a unifying framework for inspecting hidden representations of language models .\narXiv preprint arXiv:2401.06102 .\nCited by: §1 .\nGrattafiori et al. (2024)\nA. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al.\nThe llama 3 herd of models .\narXiv preprint arXiv:2407.21783 .\nCited by: §1 ,\n§4.1 .\nGreenblatt et al. (2024)\nR. Greenblatt, C. Denison, B. Wright, F. Roger, M. MacDiarmid, S. Marks, J. Treutlein, T. Belonax, J. Chen, D. Duvenaud, et al.\nAlignment faking in large language models .\narXiv preprint arXiv:2412.14093 .\nCited by: §1 ,\n§2 .\nGurnee et al. (2023)\nW. Gurnee, N. Nanda, M. Pauly, K. Harvey, D. Troitskii, and D. Bertsimas\nFinding neurons in a haystack: case studies with sparse probing .\narXiv preprint arXiv:2305.01610 .\nCited by: §5.4 .\nJain et al. (2024)\nS. Jain, R. Kirk, E. S. Lubana, R. P. Dick, H. Tanaka, E. Grefenstette, T. Rocktäschel, and D. S. Krueger\nMechanistically analyzing the effects of fine-tuning on procedurally defined tasks .\nExternal Links: 2311.12786 ,\nLink\nCited by: §2 .\nJiralerspong and Bricken (2025)\nT. Jiralerspong and T. Bricken\nCross-architecture model diffing with crosscoders: unsupervised discovery of differences between llms .\nIn Mechanistic Interpretability Workshop at NeurIPS 2025 ,\nCited by: §1 ,\n§2 ,\n§3.3 .\nKarvonen et al. (2025)\nA. Karvonen, J. Chua, C. Dumas, K. Fraser-Taliente, S. Kantamneni, J. Minder, E. Ong, A. S. Sharma, D. Wen, O. Evans, et al.\nActivation oracles: training and evaluating llms as general-purpose activation explainers .\narXiv preprint arXiv:2512.15674 .\nCited by: §2 .\nKassem et al. (2025)\nA. M. Kassem, Z. Shi, N. Rostamzadeh, and G. Farnadi\nReviving your mneme: predicting the side effects of llm unlearning and fine-tuning via sparse model diffing .\nExternal Links: 2507.21084 ,\nLink\nCited by: §2 .\nLindsey et al. (2024)\nJ. Lindsey, A. Templeton, J. Marcus, T. Conerly, J. Batson, and C. Olah\nSparse crosscoders for cross-layer features and model diffing .\nTransformer Circuits Thread .\nCited by: §1 ,\n§2 ,\n§3.1 .\nMacDiarmid et al. (2025)\nM. MacDiarmid, B. Wright, J. Uesato, J. Benton, J. Kutasov, S. Price, N. Bouscal, S. Bowman, T. Bricken, A. Cloud, et al.\nNatural emergent misalignment from reward hacking in production rl .\narXiv preprint arXiv:2511.18397 .\nCited by: §1 .\nMerchant et al. (2020)\nA. Merchant, E. Rahimtoroghi, E. Pavlick, and I. Tenney\nWhat happens to bert embeddings during fine-tuning? .\nExternal Links: 2004.14448 ,\nLink\nCited by: §2 .\nMinder et al. (2025a)\nJ. Minder, C. Dumas, C. Juang, B. Chughtai, and N. Nanda\nOvercoming sparsity artifacts in crosscoders to interpret chat-tuning .\nIn The Thirty-ninth Annual Conference on Neural Information Processing Systems ,\nCited by: §1 ,\n§2 ,\n§3.1 ,\n§4.2 .\nMinder et al. (2025b)\nJ. Minder, C. Dumas, S. Slocum, H. Casademunt, C. Holmes, R. West, and N. Nanda\nNarrow finetuning leaves clearly readable traces in activation differences .\narXiv preprint arXiv:2510.13900 .\nCited by: §1 ,\n§1 ,\n§2 ,\n§4.3 ,\n§6.2 ,\n§6.2 ,\n§6.2 ,\n§7.2 .\nMishra-Sharma et al. (2024)\nS. Mishra-Sharma, T. Bricken, J. Lindsey, A. Jermyn, J. Marcus, K. Rivoire, C. Olah, and T. Henighan\nInsights on crosscoder model diffing .\nTransformer Circuits Thread .\nCited by: §1 ,\n§2 ,\n§3.3 .\nMosbach (2023)\nM. Mosbach\nAnalyzing pre-trained and fine-tuned language models .\nIn Proceedings of the Big Picture Workshop , Y. Elazar, A. Ettinger, N. Kassner, S. Ruder, and","chars":14000,"total_chars":78005,"offset":43000,"truncated":true,"next_offset":57000}