Run step
Step 12: tool results
read_arxiv_source
Input
{
"file": "main.tex",
"offset": 0,
"arxiv_id": "2502.16681",
"max_chars": 20000
}{"ok":true,"arxiv_id":"2502.16681","file":"main.tex","kind":"text","text":"%%%%%%%% ICML 2025 EXAMPLE LATEX SUBMISSION FILE %%%%%%%%%%%%%%%%%\n\n\\documentclass{article}\n\\usepackage{hyperref}\n\n\\newcommand{\\saeneuronpedia}\n{\\mathrm{width} = 131\\mathrm{k}, \\mathrm{L0} = 114}\n\\newcommand{\\saesmall}\n{\\mathrm{width} = 16\\mathrm{k}}\n\\newcommand{\\saemid}{\\mathrm{width} = 131\\mathrm{k}}\n\\newcommand{\\saelarge}{\\mathrm{width} = 1\\mathrm{M}}\n\\newcommand{\\lzero}{\\mathrm{L0}}\n% Recommended, but optional, packages for figures and better typesetting:\n\\usepackage{microtype}\n\\usepackage{graphicx}\n\\usepackage{longtable}\n\\usepackage{subcaption}\n\\usepackage{booktabs} % for professional tables\n\n\n% hyperref makes hyperlinks in the resulting PDF.\n% If your build breaks (sometimes temporarily if a hyperlink spans a page)\n% please comment out the following usepackage line and replace\n% \\usepackage{icml2025} with \\usepackage[nohyperref]{icml2025} above.\n\\usepackage{hyperref}\n\n\n% Attempt to make hyperref and algorithmic work together better:\n\\newcommand{\\theHalgorithm}{\\arabic{algorithm}}\n\n% Use the following line for the initial blind version submitted for review:\n% \\usepackage{icml2025}\n\n% If accepted, instead use the following line for the camera-ready submission:\n\\usepackage[accepted]{arxiv}\n\n% For theorems and such\n\\usepackage{amsmath}\n\\usepackage{amssymb}\n\\usepackage{mathtools}\n\\usepackage{amsthm}\n\n% if you use cleveref..\n\\usepackage[capitalize,noabbrev]{cleveref}\n\n%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n% THEOREMS\n%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n\\theoremstyle{plain}\n\\newtheorem{theorem}{Theorem}[section]\n\\newtheorem{proposition}[theorem]{Proposition}\n\\newtheorem{lemma}[theorem]{Lemma}\n\\newtheorem{corollary}[theorem]{Corollary}\n\\theoremstyle{definition}\n\\newtheorem{definition}[theorem]{Definition}\n\\newtheorem{assumption}[theorem]{Assumption}\n\\theoremstyle{remark}\n\\newtheorem{remark}[theorem]{Remark}\n\n% Todonotes is useful during development; simply uncomment the next line\n% and comment out the line below the next line to turn off comments\n%\\usepackage[disable,textsize=tiny]{todonotes}\n\\usepackage[textsize=tiny]{todonotes}\n\n\n% new packages\n\\usepackage{xspace}\n\\usepackage{enumitem}\n\\usepackage{makecell}\n\\usepackage{graphicx}\n\n% Macros\n\\DeclareMathOperator*{\\ArgTopK}{arg\\,top\\,k}\n\n\n% The \\icmltitle you define below is probably too long as a header.\n% Therefore, a short form for the running title is supplied here:\n\\icmltitlerunning{Are Sparse Autoencoders Useful? A Case Study in Sparse Probing\n}\n\n\\begin{document}\n\n\\twocolumn[\n\\icmltitle{Are Sparse Autoencoders Useful? A Case Study in Sparse Probing}\n\n% It is OKAY to include author information, even for blind\n% submissions: the style file will automatically remove it for you\n% unless you've provided the [accepted] option to the icml2025\n% package.\n\n% List of affiliations: The first argument should be a (short)\n% identifier you will use later to specify author affiliations\n% Academic affiliations should list Department, University, City, Region, Country\n% Industry affiliations should list Company, City, Region, Country\n\n% You can specify symbols, otherwise they are numbered in order.\n% Ideally, you should not use this facility. Affiliations will be numbered\n% in order of appearance and this is the preferred way.\n\\icmlsetsymbol{equal}{*}\n\n\\begin{icmlauthorlist}\n\\icmlauthor{Subhash Kantamneni}{equal,mit}\n\\icmlauthor{Joshua Engels}{equal,mit}\n\\icmlauthor{Senthooran Rajamanoharan}{}\n\\icmlauthor{Max Tegmark}{mit}\n\\icmlauthor{Neel Nanda}{}\n\\end{icmlauthorlist}\n\n\\icmlaffiliation{mit}{Massachusetts Institute of Technology}\n\n\n\\icmlcorrespondingauthor{Subhash Kantamneni}{subhashk@mit.edu}\n\\icmlcorrespondingauthor{Joshua Engels}{jengels@mit.edu}\n\n% You may provide any keywords that you\n% find helpful for describing your paper; these are used to populate\n% the \"keywords\" metadata in the PDF but will not be shown in the document\n\\icmlkeywords{Machine Learning, ICML}\n\n\\vskip 0.3in\n]\n\n% this must go after the closing bracket ] following \\twocolumn[ ...\n\n% This command actually creates the footnote in the first column\n% listing the affiliations and the copyright notice.\n% The command takes one argument, which is text to display at the start of the footnote.\n% The \\icmlEqualContribution command is standard text for equal contribution.\n% Remove it (just {}) if you do not need this facility.\n\n%\\printAffiliationsAndNotice{} % leave blank if no need to mention equal contribution\n\\printAffiliationsAndNotice{\\icmlEqualContribution} % otherwise use the standard text.\n\n\\begin{abstract}\nSparse autoencoders (SAEs) are a popular method for interpreting concepts represented in large language model (LLM) activations. However, there is a lack of evidence regarding the validity of their interpretations due to the lack of a ground truth for the concepts used by an LLM, and a growing number of works have presented problems with current SAEs. One alternative source of evidence would be demonstrating that SAEs improve performance on downstream tasks beyond existing baselines. We test this by applying SAEs to the real-world task of LLM activation probing in four regimes: data scarcity, class imbalance, label noise, and covariate shift\\footnote{We make our code publicly available \\href{https://github.com/JoshEngels/SAE-Probes}{here.}}. Due to the difficulty of detecting concepts in these challenging settings, we hypothesize that SAEs' basis of interpretable, concept-level latents should provide a useful inductive bias. However, although SAEs occasionally perform better than baselines on individual datasets, we are unable to design ensemble methods combining SAEs with baselines that consistently outperform ensemble methods solely using baselines. Additionally, although SAEs initially appear promising for identifying spurious correlations, detecting poor dataset quality, and training multi-token probes, we are able to achieve similar results with simple non-SAE baselines as well. Though we cannot discount SAEs' utility on other tasks, our findings highlight the shortcomings of current SAEs and the need to rigorously evaluate interpretability methods on downstream tasks with strong baselines. \n% Sparse autoencoders (SAEs) have gathered significant attention in the field of mechanistic interpretability. However, there remains a lack of rigorous evidence showing their efficacy on downstream tasks, and a growing number of works have presented problems with current SAEs. In this work, we test SAE effectiveness on the well studied task of probing: learning a map from model hidden states to targets. We hypothesize that if SAEs indeed find a more interpretable or true-to-the-model basis for activations, probes trained on their latent space should offer advantages over traditional raw activation based methods. We investigate probing in multiple regimes--data scarcity, label corruption, class imbalance, and distribution shifts--and \n% % find probing regimes\n% % % aim to create the strongest possible case for SAEs, \n% % investigate multiple regimes where we hypothesize SAEs' inductive biases might help: data scarcity, label corruption, class imbalance, and distributional shifts, We find \n% find weakly positive results for SAE probes when compared to traditional activation probes. Additionally, we show through case studies on two datasets how SAE probes can identify spurious correlations and label errors. We also investigate the reasons behind the underperformance of SAEs relative to our expectations. Our findings highlight the necessity for further empirical evaluation of SAEs in real-world applications and raise the question of whether current SAEs have fundamental flaws.\n% revealing significant shortcomings in current methodologies that require rectification. \n\\end{abstract}\n\n\n\\begin{figure}\n \\centering\n \\includegraphics[width=\\linewidth]{figures/fig1.pdf}\n \\caption{SAE probes underperform the baseline of logistic regression in each regime when taking the mean across datasets. Additionally, we find that baseline methods can provide many of the interpretability insights of SAE probes.}\n \\label{fig:fig1}\n\\end{figure}\n\n\n\n\\section{Introduction}\nDictionary learning is a popular method for interpreting LLM activations \\cite{arora2018linear, yun2021transformer}; most notably, \\citet{dictionary_monosemanticity_anthropic, other_sae_paper} demonstrated the promise of sparse autoencoders (SAEs) and sparked considerable follow-up work. This includes papers focused on improving the downstream cross entropy loss of SAE reconstructions \\cite{gao2024scalingevaluatingsparseautoencoders, braun2024identifyingfunctionallyimportantfeatures, rajamanoharan2024improving,rajamanoharan2024jumpingaheadimprovingreconstruction}, improving SAE training efficiency \\cite{mudide2024efficientdictionarylearningswitch, gao2024scalingevaluatingsparseautoencoders}, finding weaknesses in SAEs and proposing solutions \\cite{chanin2024absorption, bussmann2024matryoshka}, using SAEs to understand model representation structure \\cite{engels2024languagemodelfeatureslinear, engels2024decomposing, li2024geometry}, training large suites of SAEs \\cite{gemmascope, LlamaScope}, and developing benchmarks for SAEs \\cite{huang2024ravel, karvonen2408measuring, neuronpediaSAEBenchComprehensive}.\n% Since then, the mechanistic interpretability community has published a large number of followup papers \\cite{}. Many of these papers focus on improving the performance of sparse autoencoders themselves; a popular metric is the average number of active latents versus the downstream cross entropy loss when the SAE is inserted into the model. \n\nUnfortunately, we lack a ``ground truth'' to know whether SAEs truly extract the interpretable concepts used by language models. Prior work has largely avoided this limitation by evaluating SAEs with proxy metrics like reconstruction loss \\cite{rajamanoharan2024improving, rajamanoharan2024jumpingaheadimprovingreconstruction, gao2024scalingevaluatingsparseautoencoders, olmo2024featuresmakedifferenceleveraging, braun2024identifyingfunctionallyimportantfeatures}. These metrics are tractable to optimize for, but do not necessarily align with mechanistic interpretability's (MI) goal of better understanding neural networks (see \\citet{bereska2024mechanistic} and \\citet{sharkey2025openproblemsmechanisticinterpretability} for surveys of MI). We argue that if SAEs truly advance MI's goal, they should improve performance on a real, hard-to-fake model control or explainability task.\n\nHowever, despite the extensive body of work on SAEs, there are relatively few cases where SAEs have been shown to help on such a task: \\citet{marks2024sparse} show that SAE feature circuits can help identify and remove bias from classifiers; \\citet{smith2024sparse_autoencoders} show that SAEs can identify misaligned features learned by a preference model; and \\citet{karvonen2024sieve} show that SAE feature ablations are better at preventing regex output than baselines in a setting with scarce supervised data. While these results are promising, they all study a single example in detail, and consider comparative baselines with varying levels of rigor. \n\n% \\citet{bricken2024features_classifiers} show that a probe trained on SAE latents to identify harmful bio-weapons prompts outperforms one trained on activations;\n\nAt the same time, there are also surprisingly few negative results finding that SAEs do \\textit{not} help on downstream tasks; in fact, we are only aware of \\citet{chaudhary2024evaluating}, which finds that SAE latents are worse than neurons at disentangling geographic data, and \\citet{FarrelSAEBio}, which finds that pinning related SAE latents is less effective than baselines for unlearning bioweapon knowledge. Thus, it is not clear whether SAEs are just one result away from being differentially useful, or if MI should seek fundamentally different methods. We defer a thorough examination of related work to \\cref{sec:app-related-work}. \n\n\nIn this work, we attempt to fairly evaluate the utility of SAEs by examining their competitive advantage on a concrete task: training probes from language model activations to targets \\cite{alain2016understanding}. Probing has two important qualities:\n\\vspace{-0.1cm}\n\\begin{enumerate}[topsep=1pt,itemsep=0pt,parsep=0pt,leftmargin=12pt]\n\\item \\textbf{Probing is practically useful:} Probing has been used to investigate LLM representations \\cite{space_time, othello_neel_linear, monotonic_numeric_representations}, detect safety relevant quantities \\cite{zou2023representation}, remove knowledge from models \\citep{Elazar2021}, and catch synthetic sleeper agents \\cite{macdiarmid2024sleeperagentprobes}.\n\\item \\textbf{SAEs might reasonably improve probing:} Theoretically, SAE latents are a more interpretable basis of model activations, and we hypothesize that this inductive bias will help train probes in difficult regimes. Recent work has also found some positive results for the utility of SAE probes \\cite{bricken2024features_classifiers, probingGallifant}.\n\\end{enumerate}\n\\vspace{-0.15cm}\nThus, we curate 113 linear probing datasets from a variety of settings and train linear probes on corresponding SAE latent activations (see \\cref{fig:method_diagram}). We compare to a suite of baseline methods across 4 difficult probing regimes: 1) data scarcity, 2) class imbalance, 3) label noise, and 4) co-variate shift.\n% \\vspace{-0.1cm}\n% \\begin{enumerate}[topsep=1pt,itemsep=0pt,parsep=0pt,leftmargin=20pt]\n% \\item Low data\n% \\item Label noise (concept drift)\n% \\item Out of distribution data (co-variate shift)\n% \\item Class imbalance \n% \\end{enumerate}\n% \\vspace{-0.15cm}\nUnfortunately, we find that \\textbf{SAE probes fail to offer a consistent overall advantage when added to a simulated practitioner's toolkit}. \n\n\nFurther, we explore areas that \\citet{bricken2024features_classifiers} suggest SAEs may be valuable for, including detecting attributes distributed over multiple tokens and identifying dataset issues. Although SAEs initially seemed promising in these settings, we were able to achieve the same results with improved baselines. Our results underscore the necessity for MI works to rigorously design baselines when evaluating the utility of interpretability techniques.\n\n% Further, although we found initially promising results for SAE probes in some settings, by using more rigorous baselines we concluded these were illusory. Namely, we find that \\citet{bricken2024features_classifiers}'s preliminary work demonstrating the advantage of SAE probes when applied to multiple tokens did not properly consider a multi-token comparative baseline. In our own work, we initially falsely concluded that SAE probes offered an advantage for evaluating dataset quality when baseline methods provided the same insight. \n\n% sparse probes have inductive bias towards interp, hypothesize that this interp bias can help in some settings\n% Summary of results, somewhat evidence against current SAEs being a game changer, further iteration needed, possibly larger SAEs, … \n% Check out meta SAEs paper\n% Contributions:\n\n\n% Dataset interp.\n\n\n\n\n\\begin{figure}\n \\centering\n \\includegraphics[width=\\linewidth]{figures/probing_drawio_10.pdf}\n \\caption{\\textbf{Left:} An illustration of our SAE probing method. We pass in training activation vectors from each class and train an $L_1$ regularized logistic regression probe on the latents that differ the most between classes. \\textbf{Right:} We ensure robustness of our results with the \"quiver of arrows\" approach (see \\cref{sec:quiver}): we add SAE regression into a set of methods, and see if the test accuracy of the best method (chosen by validation accuracy) increases.}\n \\label{fig:method_diagram}\n\\end{figure}\n\n\n\\begin{table*}[h]\n \\centering\n \\caption{Example probing tasks. Only tasks with short prompts shown for conciseness.}\n \\begin{tabular}{c|p{10cm}|c}\n \\hline\n \\textbf{Dataset Name} & \\textbf{Prompt} & \\textbf{Target} \\\\ \\hline\n 26\\_headline\\_isfrontpage & Sebelius’s Slow-Motion Resignation From the Cabinet. & 1 \\\\\n & MI5 References Emerge in Phone Hacking Lawsuit. & 0 \\\\ \\hline\n 36\\_sciq\\_tf & Q: Binary fission is an example of which type of production? A: asexual & 1 \\\\\n & Q: What occurs when light interacts with our atmosphere? A: prism effect & 0 \\\\ \\hline\n 114\\_nyc\\_borough\\_Manhattan & INWOOD BASKETBALL COURTS & 1 \\\\\n & PS 18 JOHN G WHITTIER & 0 \\\\ \\hline\n 149\\_twt\\_emotion\\_happiness & All moved in to our new apartment. So exciting & 1 \\\\\n & is noow calmmm eating polvoron .. yuumm & 0 \\\\ \\hline\n 155\\_athlete\\_sport\\_basketball & Jimmy Butler & 1 \\\\\n & Steve Balboni & 0\n \\\\\n \\end{tabular}\n \\label{tab:example_data}\n\\end{table*}\n\n\\section{Methodology}\n\nWe apply probes to the hidden states of two language models, Gemma-2-9B \\cite{gemma2} and Llama-3.1-8B \\cite{llama3.1-8B}. Our main paper results use Gemma-2-9B. We replicate core results on Llama-3.1-8B in \\cref{sec:app-llama}. We use JumpReLU~\\cite{rajamanoharan2024jumpingaheadimprovingreconstruction} sparse autoencoders (SAEs) from Gemma Scope \\cite{gemmascope} for Gemma-2-9B and TopK~\\cite{gao2024scalingevaluatingsparseautoencoders} SAEs from Llama Scope \\cite{LlamaScope} for Llama-3.1-8B. See \\cref{sec:app-related-work} for further background on SAEs.\n\n\\subsection{Classification Datasets}\nWe collect a diverse set of 113 binary classification datasets listed in \\cref{tab:app_all_tasks} (\\cref{sec:app_class_datasets}), including datasets that we expect to be challenging for probes.\n% We gather tasks from both academic works and more informal settings like Kaggle. \n% To avoid observational bias, we . \nFor example, 26\\_headline\\_isfrontpage requires probes to identify front-page headlines and 136\\_glue\\_mnli\\_entailment requires probes to identify logically entailment. For other example datasets, refer to \\cref{tab:example_data}. All datasets are titled in the form $\\mathrm{ID}\\_\\mathrm{description}$. We originally collected datasets for other purposes and discarded some of them; thus, while we have 113 datasets, $\\mathrm{ID}$ ranges from $[5,163]$.\n\n\nAll datasets contain $\\mathrm{prompts}$ and $\\mathrm{targets}$. Probes are tasked with predicting the $\\mathrm{target}$ from the model's hidden activations when run on a $\\mathrm{prompt}$. We focus on binary classification, since SAEs are mostly thought of as representing binarized latents (see \\citet{dictionary_monosemanticity_anthropic}). Thus, $\\mathrm{target}$ is either $0$ or $1$. The prompts in our datasets range in length from 5 tokens to a (left-truncated) maximum of 1024 tokens. \n\n\n% e.g. automated-interpretability methods just seek to predict whether or not a latent fires,\n\n\\subsection{Probing Strategy}\n\\label{sec:probing_strat}\nTo train a probe on a given dataset, we first run the model on all $\\mathrm{prompts}$ from that dataset to generate model activations at each layer $l$ on the last token, $X_{-1}^l$.$X_{-1}^l$ is of shape $\\mathrm{(len(prompts), model\\_dim)}$. We probe the last token to ensure the target information was present in the preceding context. For activation probes, we then train a probe $p$ to map from $X_{-1}^l \\mapsto t$, where $t$ are the $\\mathrm{targets}$ in our dataset. \n\n\nOur SAE probe training technique is summarized in \\cref{fig:method_diagram}. We first pass the training dataset $X_{-1}^l$ through the SAE encoder, resulting in a batch of vectors in the SAE latent space $Z = \\mathrm{SAE}(X_{-1}^l)$ of shape $(\\mathrm{len(prompts)}, W)$, where $W$ is the width of the SAE. We do not train probes directly on the SAE latent space because we hypothesize that a small number of SAE latents encodes the desired concept. Instead, we create a basis of latents with the highest average absolute difference between the set of training prompts $T_1$ with $\\mathrm{target} = 1$ and the set of training prompts $T_0$ with $\\mathrm{target} = 0$. More formally, if $\\mathrm{SAE}(X_{-1}^l) \\in \\mathbb{R}^{W}$, where $ W \\gg d_\\textrm{mode","chars":20000,"total_chars":104886,"offset":0,"truncated":true,"next_offset":20000}