Step 6: tool results

provenance_read_source

{
  "offset": 36000,
  "max_chars": 14000,
  "source_id": "b45014e2-63ec-41af-8a6f-59db01b1e0fc"
}

{"success":true,"source":{"id":"b45014e2-63ec-41af-8a6f-59db01b1e0fc","url":"https://arxiv.org/pdf/2502.16681","title":"Are Sparse Autoencoders Useful? A Case Study in Sparse Probing","source_type":"preprint"},"origin":"stored","total_chars":97632,"offset":36000,"content":"\nOctober 4, 2023\nReLU (updated)\nConerly et al. (2024)\nApril 26, 2024\nGated\nRajamanoharan et al. (2024a)\nMay 1, 2024\nTopK\nGao et al. (2024)\nJune 6, 2024\nJumpReLU\nRajamanoharan et al. (2024b)\nJuly 19, 2024\nBatchTopK\nBussmann et al. (2024a)\nJuly 19, 2024\np-annealing\nKarvonen et al.\nJuly 31, 2024\nMatryoshka\nBussmann et al. (2024b)\nDecember 19, 2024\nseems to be a possible slight improvement with newer SAE\narchitectures.\n7. Conclusion\nOur evaluation of sparse autoencoders (SAEs) reveals funda-\nmental limitations in current SAE methodologies. Despite\nexpectations that interpretable SAE latents would provide\na competitive advantage in probing, we found no improve-\nment over traditional methods across multiple regimes and\nover 100 datasets. Some failures expose critical weaknesses\n– for instance, SAE latents struggle to be robust to distribu-\ntional shifts and fail to represent complex concepts – while\nothers are more mundane – lower L0 SAEs create worse\nsparse probes. More broadly, our findings highlight the need\nfor the field of mechanistic interpretability to evaluate tech-\nniques with rigorous baselines. While prior work reported\nadvantages for SAE probes, our robust quiver of arrows\nmethodology, use of stronger baselines, and evaluation on\na large set of datasets demonstrate these advantages to be\nillusory. Even in our own analysis, initial conclusions favor-\ning SAE interpretability were later overturned when proper\nbaselines were considered. We do not consider this work\nto be a wholesale critique of the SAE paradigm. Instead,\nwe view it as a single rigorously evaluated datapoint to\ncontextualize SAE utility and to motivate a more thorough\nexamination of methods in mechanistic interpretability.\nLimitations We study the performance of SAE probes as a\nproxy for SAE utility. While we believe that probing perfor-\nmance is a more effective measurement than typical SAE\nevaluations like downstream cross-entropy loss or recon-\nstruction error, it is ultimately still a proxy metric. We chose\nto study difficult probing regimes to try to find cases whereReLU (orig.)\nReLU (new)\nGated\nTopK\nJump ReLU\nBatch TopK\np-anneal\nMatryoshka\nArchitecture (In Order of Release)\n0.20\n0.15\n0.10\n0.05\n0.00\nSAE - Baseline AUC\nAUC for SAE Architecture vs Baseline\nk=16 at Best L0 in Standard Conditions\nFigure 12. We see that there is a slight uptick in probing perfor-\nmance compared to baselines in standard conditions with more\nrecent architectures. However, the spread of the data is consider-\nably larger than this improvement.\nthe inductive bias of SAE latents outweighed imperfect SAE\nreconstruction, but it is possible that even a basis of “true”\nmodel representations would offer only a mild inductive\nbias advantage over traditional linear probing. We are thus\nexcited for future work studying probing on toy models (e.g.\nmodels trained on board games (Karvonen et al., 2024c))\nwhere the “true” model features are known.\nFinally, it is possible that further optimization of the SAE\nprobe baseline might increase performance such that it beats\nbaseline methods. For example, Gallifant et al. (2025) find\nthat SAE probing with a number of modifications (multi-\ntoken probes, binarization of latents, and probing with k =\nfull SAE dimension) beats baseline methods on safety-\nrelevant datasets (albeit only comparing to single-token\nbaseline probes). However, we believe a main takeaway\nof our paper is that equivalent effort should be put into\noptimizing baselines. Indeed, we examine the effect of\nbinarizing latents in Appendix J and find that it does not\nsignificantly help.\n8. Acknowledgments\nThis work was conducted as part of the ML Alignment\n& Theory Scholars (MATS) Program. We would like to\nthank the members of the Tegmark group and Neel Nanda’s\nMATS stream for helpful comments and feedback. JE was\nsupported by the NSF Graduate Research Fellowship (Grant\nNo. 2141064).\n9\n\nAre Sparse Autoencoders Useful? A Case Study in Sparse Probing\n9. Author Contributions\nJE and SK jointly led the project and wrote the paper. JE\ncame up with the Quiver of Arrows approach, ran the SAE\nmethods, discovered the initial GLUE CoLA and AI vs. Hu-\nman results, discovered Gemma Scope SAE latent reliance\non language, and ran multi-token experiments. SK cleaned\na majority of the datasets, ran baseline methods, did the\nprobe interpretability experiments, ran the main text covari-\nate shift experiments, did follow-up experiments on dataset\nerrors and latent interpretability, and made most of the final\nmain body plots. SR and MT provided advice, guidance,\nand general mentorship, and edited the paper. NN was the\nprimary senior author of the project and provided leadership,\ndirection, and advice throughout.\nReferences\nAdebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt,\nM., and Kim, B. Sanity checks for saliency maps. Ad-\nvances in neural information processing systems, 31,\n2018.\nAI, T. and Ishii, D. Spam Text Message\nClassification — kaggle.com. https:\n//www.kaggle.com/datasets/team-ai/\nspam-text-message-classification. [Ac-\ncessed 21-01-2025].\nAlain, G. Understanding intermediate layers using linear\nclassifier probes. arXiv preprint arXiv:1610.01644, 2016.\nAllenAI. allenai/basic arithmetic · Datasets at Hugging Face\n— huggingface.co. https://huggingface.co/\ndatasets/allenai/basic_arithmetic. [Ac-\ncessed 21-01-2025].\nAnthropic. Introducing computer use, a new\nClaude 3.5 Sonnet, and Claude 3.5 Haiku, 2024.\nURL https://www.anthropic.com/news/\n3-5-models-and-computer-use. Accessed:\n2025-01-30.\nArora, S., Li, Y., Liang, Y., Ma, T., and Risteski, A. Linear\nalgebraic structure of word senses, with applications to\npolysemy. Transactions of the Association for Computa-\ntional Linguistics, 6:483–495, 2018.\nBen Zhou, Daniel Khashabi, Q. N. and Roth, D. “going on\na vacation” takes longer than “going for a walk”: A study\nof temporal commonsense understanding. In EMNLP,\n2019.\nBereska, L. and Gavves, E. Mechanistic interpretability\nfor ai safety–a review. arXiv preprint arXiv:2404.14082,\n2024.\nBills, S., Cammarata, N., Mossing, D., Tillman, H.,\nGao, L., Goh, G., Sutskever, I., Leike, J., Wu,\nJ., and Saunders, W. Language models can\nexplain neurons in language models. https:\n//openaipublic.blob.core.windows.net/\nneuron-explainer/paper/index.html,\n2023.\nBisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y.\nPiqa: Reasoning about physical commonsense in natural\nlanguage. In Thirty-Fourth AAAI Conference on Artificial\nIntelligence, 2020.\nBolukbasi, T., Pearce, A., Yuan, A., Coenen, A., Reif, E.,\nVi´egas, F., and Wattenberg, M. An interpretability illusion\nfor bert. arXiv preprint arXiv:2104.07143, 2021.\nBraun, D., Taylor, J., Goldowsky-Dill, N., and Sharkey,\nL. Identifying functionally important features with end-\nto-end sparse dictionary learning, 2024. URL https:\n//arxiv.org/abs/2405.12241.\nBricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A.,\nConerly, T., Turner, N., Anil, C., Denison, C., Askell, A.,\nLasenby, R., Wu, Y., Kravec, S., Schiefer, N., Maxwell,\nT., Joseph, N., Hatfield-Dodds, Z., Tamkin, A., Nguyen,\nK., McLean, B., Burke, J. E., Hume, T., Carter, S.,\nHenighan, T., and Olah, C. Towards monosemanticity:\nDecomposing language models with dictionary learning.\nTransformer Circuits Thread, 2023. https://transformer-\ncircuits.pub/2023/monosemantic-features/index.html.\nBricken, T., Marcus, J., Mishra-Sharma, S., Tong,\nM., Perez, E., Sharma, M., Rivoire, K., and\nHenighan, T. Using dictionary learning features\nas classifiers, October 2024a. URL https:\n//transformer-circuits.pub/2024/\nfeatures-as-classifiers/index.html.\nAccessed: 2025-01-23.\nBricken, T., Marcus, J., Mishra-Sharma, S., Tong,\nM., Perez, E., Sharma, M., Rivoire, K., and\nHenighan, T. Using dictionary learning features\nas classifiers, October 2024b. URL https:\n//transformer-circuits.pub/2024/\nfeatures-as-classifiers/index.html.\nTransformer Circuits.\nBussmann, B., Leask, P., and Nanda, N. Batchtopk sparse\nautoencoders, 2024a.\nBussmann, B., Leask, P., and Nanda, N.\nLearning multi-level features with ma-\ntryoshka saes. https://www.lesswrong.\ncom/posts/rKM9b6B2LqwSB5ToN/\nlearning-multi-level-features-with-matryoshka-sae\n2024b. Accessed: 2025-01-23.\n10\n\nAre Sparse Autoencoders Useful? A Case Study in Sparse Probing\nChalnev, S., Siu, M., and Conmy, A. Improving steering\nvectors by targeting sparse autoencoder features. arXiv\npreprint arXiv:2411.02193, 2024.\nChanin, D., Wilken-Smith, J., Dulka, T., Bhatnagar, H., and\nBloom, J. A is for absorption: Studying feature splitting\nand absorption in sparse autoencoders. arXiv preprint\narXiv:2409.14507, 2024.\nChaudhary, M. and Geiger, A. Evaluating open-source\nsparse autoencoders on disentangling factual knowledge\nin gpt-2 small. arXiv preprint arXiv:2409.04478, 2024.\nChen, T. and Guestrin, C. Xgboost: A scalable tree\nboosting system. In Proceedings of the 22nd ACM\nSIGKDD International Conference on Knowledge Dis-\ncovery and Data Mining, KDD ’16, pp. 785–794,\nNew York, NY, USA, 2016. Association for Comput-\ning Machinery. ISBN 9781450342322. doi: 10.1145/\n2939672.2939785. URL https://doi.org/10.\n1145/2939672.2939785.\nclmentbisaillon. fake-and-real-news-dataset\n— kaggle.com. https://www.kaggle.\ncom/datasets/clmentbisaillon/\nfake-and-real-news-dataset. [Accessed\n21-01-2025].\nConerly, T., Templeton, A., Bricken, T., Mar-\ncus, J., and Henighan, T. Circuits Updates -\nApril 2024 — transformer-circuits.pub. https:\n//transformer-circuits.pub/2024/\napril-update/index.html#training-saes,\n2024. [Accessed 15-02-2025].\nCunningham, H., Ewart, A., Riggs, L., Huben, R., and\nSharkey, L. Sparse autoencoders find highly inter-\npretable features in language models. arXiv preprint\narXiv:2309.08600, 2023.\nDavidson, T., Warmsley, D., Macy, M., and Weber, I. Auto-\nmated hate speech detection and the problem of offensive\nlanguage. In Proceedings of the 11th International AAAI\nConference on Web and Social Media, ICWSM ’17, pp.\n512–515, 2017.\nDublish, T. Text classification documen-\ntation — kaggle.com. https://www.\nkaggle.com/datasets/tanishqdublish/\ntext-classification-documentation.\n[Accessed 21-01-2025].\nElazar, Y., Ravfogel, S., Jacovi, A., and Goldberg, Y. Am-\nnesic probing: Behavioral explanation with amnesic coun-\nterfactuals. Transactions of the Association for Com-\nputational Linguistics, 9:160–175, March 2021. ISSN\n2307-387X. doi: 10.1162/tacl a 00359. URL http:\n//dx.doi.org/10.1162/tacl_a_00359.\nEngels, J., Michaud, E. J., Liao, I., Gurnee, W., and\nTegmark, M. Not all language model features are lin-\near, 2024a. URL https://arxiv.org/abs/2405.\n14860.\nEngels, J., Riggs, L., and Tegmark, M. Decomposing\nthe dark matter of sparse autoencoders. arXiv preprint\narXiv:2410.14670, 2024b.\nFalgunipatel19. Medical Text Dataset -Cancer Doc\nClassification — kaggle.com. https://www.\nkaggle.com/datasets/falgunipatel19/\nbiomedical-text-publication-classification.\n[Accessed 21-01-2025].\nFarrell, E., Lau, Y.-T., and Conmy, A. Applying sparse\nautoencoders to unlearn knowledge in language models,\n2024.\nGaggar, R., Bhagchandani, A., and Oza, H. Machine-\ngenerated text detection using deep learning, 2023.\nGallifant, J., Chen, S., Sasse, K., Aerts, H., Hartvigsen, T.,\nand Bitterman, D. S. Sparse autoencoder features for\nclassifications and transferability, 2025.\nGao, L., Biderman, S., Black, S., Golding, L., Hoppe, T.,\nFoster, C., Phang, J., He, H., Thite, A., Nabeshima, N.,\nPresser, S., and Leahy, C. The Pile: An 800gb dataset\nof diverse text for language modeling. arXiv preprint\narXiv:2101.00027, 2020.\nGao, L., la Tour, T. D., Tillman, H., Goh, G., Troll, R.,\nRadford, A., Sutskever, I., Leike, J., and Wu, J. Scaling\nand evaluating sparse autoencoders, 2024. URL https:\n//arxiv.org/abs/2406.04093.\nGerami, S. AI Vs Human Text — kaggle.com.\nhttps://www.kaggle.com/datasets/\nshanegerami/ai-vs-human-text/data.\n[Accessed 21-01-2025].\nGoh, A. IT Service Ticket Classification\nDataset — kaggle.com. https://www.\nkaggle.com/datasets/adisongoh/\nit-service-ticket-classification-dataset.\n[Accessed 21-01-2025].\nGrattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian,\nA., et al. The llama 3 herd of models, 2024. URL https:\n//arxiv.org/abs/2407.21783.\nGupta, P. Emotion Detection from Text\n— kaggle.com. https://www.kaggle.\ncom/datasets/pashupatigupta/\nemotion-detection-from-text. [Accessed\n21-01-2025].\n11\n\nAre Sparse Autoencoders Useful? A Case Study in Sparse Probing\nGurnee, W. and Tegmark, M. Language models represent\nspace and time. arXiv preprint arXiv:2310.02207, 2023.\nGurnee, W. and Tegmark, M. Language models represent\nspace and time. In The Twelfth International Conference\non Learning Representations, 2024. URL https://\nopenreview.net/forum?id=jE8xbmvFin.\nGurnee, W., Nanda, N., Pauly, M., Harvey, K., Troit-\nskii, D., and Bertsimas, D. Finding neurons in a\nhaystack: Case studies with sparse probing. Transac-\ntions on Machine Learning Research, 2023. ISSN 2835-\n8856. URL https://openreview.net/forum?\nid=JYs1R9IMJr.\nHe, Z., Shu, W., Ge, X., Chen, L., Wang, J., Zhou, Y., Liu,\nF., Guo, Q., Huang, X., Wu, Z., Jiang, Y.-G., and Qiu,\nX. Llama scope: Extracting millions of features from\nllama-3.1-8b with sparse autoencoders, 2024.\nHeap, T., Lawson, T., Farnik, L., and Aitchison, L. Sparse\nautoencoders can interpret randomly initialized transform-\ners, 2025. URL https://arxiv.org/abs/2501.\n17727.\nHeinzerling, B. and Inui, K. Monotonic representation of\nnumeric properties in language models. arXiv preprint\narXiv:2403.10381, 2024.\nHendrycks, D., Burns, C., Basart, S., Critch, A., Li, J., Song,\nD., and Steinhardt, J. Aligning ai with shared human\nvalues. Proceedings of the International Conference on\nLearning Representations (ICLR), 2021.\nHuang, J., Wu, Z., Potts, C., Geva, M., and Geiger, A.\nRavel: Evaluating interpretability methods on disentan-\ngling language model representations. arXiv preprint\narXiv:2402.17700, 2024.\nJohannes Welbl, Nelson F. Liu, M. G. Crowdsourcing mul-\ntiple choice science questions. 2017.\nKarvonen, A., Wright, B., Rager, C., Angell, R., Bri","truncated":true,"next_offset":50000,"note":"More text follows; call again with offset 50000 to continue."}