Step 6: tool results

provenance_read_source

{
  "offset": 20000,
  "max_chars": 16000,
  "source_id": "ff9222f2-759d-4782-a7ff-8c811f759780"
}

{"success":true,"source":{"id":"ff9222f2-759d-4782-a7ff-8c811f759780","url":"https://arxiv.org/pdf/2605.31183","title":"Steering LLMs? Actually, Sparse Autoencoders can outperform simple baselines","source_type":"preprint"},"origin":"stored","total_chars":55411,"offset":20000,"content":" early experimental phase for Gemma-2-9b to reside in the second half of the\nmodel, anticipating reasonable steering performance. We arbitrarily selected layer 17 to examine whether\nour method would also generalize to earlier layers. Layer 17 was excluded from method development and\nparameter tuning to ensure a reliable result.\nParameter tuning. Since our pipeline contains a set of parameters (π0, K, support threshold, and output\nscore threshold), we use a hold-out approach to tuning. Here, we exclusively tune using layer 32 and the\n2For each split, we select α by maximizing over the training-partition and report the score on the evaluation-partition\n3We enumerate the layers using the original indices from Lieberum et al. (2024) starting from zero.\n6\n\nfora: academia, biology, history, law, literature, physics, and sports. Hence, the remaining fora and layer 17\nare not included. As elaborated in section 3.1, K and the output score threshold were chosen heuristically\nin the early experimentation in this setup. We find that pipeline performance seems particularly sensitive to\nthe choices of π0 and the support threshold. Since it is infeasible to evaluate it on the full Cartesian product\nof features × labels, we use a manual iterative approach. Here, we select sensible initial values and evaluate\nthe assigned labels. We then inspect the results, tune the parameters optimizing for concept_rating, and\nrepeat. After a few iterations, we converge at π0 = 10−3 and a support threshold of 50, still being sensible\nvalues.\nBaseline. As baselines, we use those provided in Wu et al. (2025), including a reproduction of the SAE\nand prompt baselines named NP random. NP random samples 100 features at random and evaluates their\nofficial Neuronpedia labels for steering and prompting.\nInterpretation. Comparisons of steering performances between feature-steering and prompt-steering are\nshown in figures 2 and 6 (see appendix B). In figure 6, we see that our proposed pipeline is indeed capable of\nassigning correct labels to the causal effects of features. While we observe some variability between different\nfora, we mostly observe similar effects when comparing feature-steering to prompt-steering.\nFigure 2 depicts the overall steering performance and shows a general improvement over those reported in Wu\net al. (2025). We observe that multiple fora provide labels that result in feature-steering performance on par\nwith the LoRA reference. We also observe a clear improvement in the prompt-steering performance, possibly\ndue to the subject model’s better understanding of simpler labels. The reference Neuronpedia performance\nis largely consistent with our results.\nWhen considering the performance increase of both feature-steering and prompt-steering from the Neuron-\npedia reference, one could be tempted to alternatively suggest better label comprehension by the LLM judge\nas an explanation. However, when considering the concept_rating performance in figure 6, we now observe\na mostly closed gap between feature-steering and prompt-steering compared to the Neuronpedia baseline.\nSince concept_rating is the only component in AxBench that relies on the feature-label, the alternative\nexplanation does not fully account for the results.\nAblation. We perform an ablation study to examine the impacts of the roles of 1) a high sparsity, and\n2) the output score. The results are shown in figures 7 and 8. Here we see that using high sparsity SAEs\n(i.e., low ℓ0) seemingly does not greatly impact the steering performance. We also observe a rather modest\nperformance improvement caused by the output score, in both the low and high sparsity SAEs.\nTo examine the role of the output score, we compare randomly selected features with features selected by\nour pipeline. In both feature sets, we fetch their original Neuronpedia labels. Here, we observe better\nfeature-steering performance of those selected by the pipeline, even without using the output score.\n5 Discussion\nWe find the results of section 4 somewhat surprising for the two reasons: 1) causality generally cannot be\ntaken for granted given a correlation/dependence, yet removing the causal-component of the pipeline (the\noutput-score) does not greatly affect its performance; and 2) the reported feature-steering performance in\nWu et al. (2025) was much weaker compared to our setup.\nLinear Representation Hypothesis. While Wu et al. (2025) reported obtaining rather poor steering per-\nformance on all methods that linearly steer, we found that SAEs can, surprisingly, steer with a performance\nclose to par with LoRA that uses adapter weights (Hu et al., 2022). This finding demonstrates the power of\nusing linear methods for understanding the internals of language models.\nAxBench. Since AxBench uses an LLM-judge to rate steered generations, we find it natural to ask whether\nthe LLM-judge could exhibit a bias to favor some steering methods differently than a population of human\njudges. To investigate this, we conducted a small trial with Danish bachelor students and found indications\nthat they would generally rate some scores more optimistically than GPT-4o-mini, however, we did not find\nclear evidence of a bias in the trial. We note that the trial had a small sample size, so a more thorough user\nstudy might still be relevant. However, while inspecting our pipeline evaluation results, we observed multiple\n7\n\nFigure 2: AxBench aggregated ratings for two layers of Gemma-2-9b-it across multiple stackexchanges. Here the Gemma-Scope SAEs have width\n131k, are inserted into the residual stream, and have sparsity parameters ℓ0 ≈ 11 and ℓ0 ≈ 10 respectively (highest sparsity). The gray datapoints\nillustrate the average performances of individual selected label, where the jitter on the x-axis is illustrative of the density estimated via KDE. The\nconfidence intervals were computed using non-parametric bootstrapping. The figure also shows multiple baselines: NP random being 100 randomly\nselected feature-label pairs from Neuronpedia, and multiple performances from Wu et al. (2025). The Neuronpedia baselines are based on the higher\nℓ0 counterparts of the selected SAEs.\n8\n\nexamples of a label semantically adjacent to, or even fully matching, the elicited topic, where AxBench still\nassigned it a concept score of 0. We included some of these examples in appendix D. Since AxBench relies\non simply prompting the LLM judge with a detailed instruction without examples, it seems natural to ask\nwhether a few-shot prompting approach could benefit its accuracy and alignment.\nApplications. Although our results suggest an improved feature-steering performance compared to Wu\net al. (2025), it is important to acknowledge that prompting is a baseline that is likely hard to match,\nas the model was trained to respond to prompts. In our view, feature-steering is an exercise in controlled\ndamage of the subject model, but it is a damage nonetheless, rendering it less useful in production. However,\nin a research setting, the causal nature of features can be reassuring that an assigned label is indeed the\nground truth, and not simply an artifact of an idiosyncratic dataset (Bolukbasi et al., 2021). Hence, Sparse\nAutoencoders could still be of interest for interpretability and explainability researchers. Our proposed\nsupervised pipeline also has the advantage of being applicable outside the text-domain, potentially enabling\nresearchers to explore the internals of models in computer vision, EEG-data, etc.\nInfluence of support. Since we found it beneficial to use a relatively high support threshold, it would\nbe reasonable to reflect on its effects on the setup. While its primary motivation was to reduce variance in\nthe calibrated F1 score (a direct effect), it likely also indicates the general popularity of a topic outside the\nrespective Stack Exchange forum. Since it is not public knowledge whether Gemma-Scope was trained on\nStack Exchange, we can only speculate that topics of higher general popularity may also more frequently\noccur in its training set. As the reconstruction loss will likely favor more frequent topics to be learned as\nfeatures (Muhamed et al., 2025), the high support threshold could filter out unlikely candidate labels as an\nindirect effect.\n6 Conclusion\nSparse Autoencoders initially seemed promising for exploring the internals of LLMs, and gained traction\nfollowing publications such as Bricken et al. (2023); Templeton et al. (2024); Lieberum et al. (2024) and\nmore. However, the optimism around the model has since faded, and its reliability has been cast into\nquestion. In this work, we have demonstrated that Sparse Autoencoders can, to an extent, live up to the\noriginal hype, even if they are not the panacea of interpretability and steering once thought to be. We\nfind that interpretable features of SAEs exhibit a surprisingly causal impact on the model predictions when\nthe labels are selected by a different approach than the one used to construct the Neuronpedia labels. Our\nproposed labelling method allows Sparse Autoencoders to perform on par with the LoRA baseline provided\nin Wu et al. (2025), vastly outperforming their reported capabilities. We further see indications that both\nlow- and high-sparsity Sparse Autoencoders can be leveraged for steering tasks.\nBroader Impact Statement\nThis work explores Sparse Autoencoders for model steering and interpretability. We believe the direct\nethical and societal implications of our work are minimal, as it primarily contributes to an understanding of\nlanguage models. In a broader perspective, we believe this could be useful for performing safety auditing,\nmodel debugging, and alignment research. We acknowledge the potential risk associated with enabling\ndirect manipulation of the open source models that could enable the expression of possessed dangerous\ncapabilities. However, we consider the additional risk posed by our framework insignificant compared to\nstandard finetuning approaches.\nAcknowledgements\nSpecial thanks to Hiba Nassar at DTU Cognitive Systems for letting us run the user study in her course\n\"02462 Signals and Data\" at DTU Compute in Autumn 2025. This work was supported by the Novo Nordisk\nFoundation grant NNF22OC0076907 “Cognitive spaces - Next generation explainability” and by the Pioneer\nCentre for AI, DNRF grant number P1.\n9\n\nReferences\nGuillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes, 2018.\nURL https://arxiv.org/abs/1610.01644.\nDana Arad, Aaron Mueller, and Yonatan Belinkov. Saes are good for steering – if you select the right\nfeatures, 2025. URL https://arxiv.org/abs/2505.20063.\nTolga Bolukbasi, Adam Pearce, Ann Yuan, Andy Coenen, Emily Reif, Fernanda Viégas, and Martin Wat-\ntenberg. An interpretability illusion for bert. arXiv preprint arXiv:2104.07143, 2021.\nTrenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner,\nCem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer,\nTim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean,\nJosiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monose-\nmanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023.\nhttps://transformer-circuits.pub/2023/monosemantic-features/index.html.\nRyan Cotterell, Anej Svete, Clara Meister, Tianyu Liu, and Li Du. Formal aspects of language modeling,\n2024. URL https://arxiv.org/abs/2311.04329.\nJonathan Crabbé and Mihaela van der Schaar. Concept activation regions: A generalized framework for\nconcept-based explanations. Advances in Neural Information Processing Systems, 35:2590–2607, 2022.\nHoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find\nhighly interpretable features in language models. arXiv preprint arXiv:2309.08600, 2023.\nEleutherAI. stackexchange_dataset. https://github.com/EleutherAI/stackexchange-dataset, 2020.\nAccessed: 2025-11-04.\nNelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell,\nYuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Trans-\nformer Circuits Thread, 1(1):12, 2021.\nNelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac\nHatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan,\nDario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer\nCircuits Thread, 2022. URL https://transformer-circuits.pub/2022/toy_model/index.html.\nN Benjamin Erichson, Zhewei Yao, and Michael W Mahoney. Jumprelu: A retrofit defense strategy for\nadversarial attacks. arXiv preprint arXiv:1904.03750, 2019.\nEoin Farrell, Yeu-Tong Lau, and Arthur Conmy. Applying sparse autoencoders to unlearn knowledge in\nlanguage models, 2024. URL https://arxiv.org/abs/2410.19278.\nMoghis Fereidouni, Muhammad Umair Haider, Peizhong Ju, and A. B. Siddique. Evaluating sparse autoen-\ncoders for monosemantic representation, 2025. URL https://arxiv.org/abs/2508.15094.\nLeo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan\nLeike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093,\n2024.\nAaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-\nDahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, and Angela Fan et al.\nThe llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783.\nWes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas.\nFinding neurons in a haystack: Case studies with sparse probing. arXiv preprint arXiv:2305.01610, 2023.\n10\n\nStefan Haufe, Frank Meinecke, Kai Görgen, Sven Dähne, John-Dylan Haynes, Benjamin Blankertz, and\nFelix Bießmann. On the interpretation of weight vectors of linear models in multivariate neuroimaging.\nNeuroImage, 87:96–110, 2014. ISSN 1053-8119. doi: https://doi.org/10.1016/j.neuroimage.2013.10.067.\nURL https://www.sciencedirect.com/science/article/pii/S1053811913010914.\nZhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang, Yunhua Zhou, Frances Liu, Qipeng\nGuo, Xuanjing Huang, Zuxuan Wu, Yu-Gang Jiang, and Xipeng Qiu. Llama scope: Extracting millions of\nfeatures from llama-3.1-8b with sparse autoencoders, 2024. URL https://arxiv.org/abs/2410.20526.\nZirui He, Haiyan Zhao, Yiran Qiao, Fan Yang, Ali Payani, Jing Ma, and Mengnan Du. Saif: A sparse\nautoencoder framework for interpreting and steering instruction following of language models. arXiv\npreprint arXiv:2502.11356, 2025.\nEdward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu\nChen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.\nBeen Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory Sayres.\nInterpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav),\n2018. URL https://arxiv.org/abs/1711.11279.\nXuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and\nTatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. https:\n//github.com/tatsu-lab/alpaca_eval, 5 2023.\nTom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma,\nJános Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders\neverywhere all at once on gemma 2. arXiv preprint arXiv:2408.05147, 2024.\nAleksandar Makelov, George Lange, and Neel Nanda. Towards principled evaluations of sparse autoencoders\nfor interpretability and control. arXiv preprint arXiv:2405.08366, 2024.\nAashiq Muhamed, Mona Diab, and Virginia Smith. Decoding dark matter: Specialized sparse autoencoders\nfor interpret","truncated":true,"next_offset":36000,"note":"More text follows; call again with offset 36000 to continue."}