Are Sparse Autoencoders Useful? A Case Study in Sparse Probing

Are Sparse Autoencoders Useful? A Case Study in Sparse Probing Subhash Kantamneni * 1 Joshua Engels * 1 Senthooran Rajamanoharan Max Tegmark 1 Neel Nanda Abstract Sparse autoencoders (SAEs) are a popular method for interpreting concepts represented in large lan- guage model (LLM) activations. However, there is a lack of evidence regarding the validity of their interpretations due to the lack of a ground truth for the concepts used by an LLM, and a grow- ing number of works have presented problems with current SAEs. One alternative source of evi- dence would be demonstrating that SAEs improve performance on downstream tasks beyond exist- ing baselines. We test this by applying SAEs to the real-world task of LLM activation probing in four regimes: data scarcity, class imbalance, label noise, and covariate shift1. Due to the difficulty of detecting concepts in these challenging settings, we hypothesize that SAEs’ basis of interpretable, concept-level latents should provide a useful in- ductive bias. However, although SAEs occasion- ally perform better than baselines on individual datasets, we are unable to design ensemble meth- ods combining SAEs with baselines that consis- tently outperform ensemble methods solely using baselines. Additionally, although SAEs initially appear promising for identifying spurious correla- tions, detecting poor dataset quality, and training multi-token probes, we are able to achieve simi- lar results with simple non-SAE baselines as well. Though we cannot discount SAEs’ utility on other tasks, our findings highlight the shortcomings of current SAEs and the need to rigorously evalu- ate interpretability methods on downstream tasks with strong baselines. 1. Introduction Dictionary learning is a popular method for interpreting LLM activations (Arora et al., 2018; Yun et al., 2021); most *Equal contribution 1Massachusetts Institute of Technology. Correspondence to: Subhash Kantamneni <subhashk@mit.edu>, Joshua Engels <jengels@mit.edu>. 1We make our code publicly available here.Standard Conditions Data Scarcity Class Imbalance Label Noise OOD Interp. Do SAE Probes Help? N/A ~ Figure 1. SAE probes underperform the baseline of logistic re- gression in each regime when taking the mean across datasets. Additionally, we find that baseline methods can provide many of the interpretability insights of SAE probes. notably, Bricken et al. (2023); Cunningham et al. (2023) demonstrated the promise of sparse autoencoders (SAEs) and sparked considerable follow-up work. This includes papers focused on improving the downstream cross entropy loss of SAE reconstructions (Gao et al., 2024; Braun et al., 2024; Rajamanoharan et al., 2024a;b), improving SAE train- ing efficiency (Mudide et al., 2024; Gao et al., 2024), finding weaknesses in SAEs and proposing solutions (Chanin et al., 2024; Bussmann et al., 2024b), using SAEs to understand model representation structure (Engels et al., 2024a;b; Li et al., 2024), training large suites of SAEs (Lieberum et al., 2024; He et al., 2024), and developing benchmarks for SAEs (Huang et al., 2024; Karvonen et al.; 2024b). Unfortunately, we lack a “ground truth” to know whether SAEs truly extract the interpretable concepts used by lan- guage models. Prior work has largely avoided this limitation by evaluating SAEs with proxy metrics like reconstruction loss (Rajamanoharan et al., 2024a;b; Gao et al., 2024; Olmo et al., 2024; Braun et al., 2024). These metrics are tractable to optimize for, but do not necessarily align with mecha- nistic interpretability’s (MI) goal of better understanding neural networks (see Bereska & Gavves (2024) and Sharkey et al. (2025) for surveys of MI). We argue that if SAEs truly advance MI’s goal, they should improve performance on a real, hard-to-fake model control or explainability task. However, despite the extensive body of work on SAEs, there are relatively few cases where SAEs have been shown to 1 arXiv:2502.16681v1 [cs.LG] 23 Feb 2025

Are Sparse Autoencoders Useful? A Case Study in Sparse Probing help on such a task: Marks et al. (2024) show that SAE feature circuits can help identify and remove bias from classifiers; Smith & Brinkmann (2024) show that SAEs can identify misaligned features learned by a preference model; and Karvonen et al. (2024a) show that SAE feature ablations are better at preventing regex output than baselines in a setting with scarce supervised data. While these results are promising, they all study a single example in detail, and consider comparative baselines with varying levels of rigor. At the same time, there are also surprisingly few negative results finding that SAEs do not help on downstream tasks; in fact, we are only aware of Chaudhary & Geiger (2024), which finds that SAE latents are worse than neurons at dis- entangling geographic data, and Farrell et al. (2024), which finds that pinning related SAE latents is less effective than baselines for unlearning bioweapon knowledge. Thus, it is not clear whether SAEs are just one result away from being differentially useful, or if MI should seek fundamentally dif- ferent methods. We defer a thorough examination of related work to Appendix A. In this work, we attempt to fairly evaluate the utility of SAEs by examining their competitive advantage on a concrete task: training probes from language model activations to targets (Alain, 2016). Probing has two important qualities: 1. Probing is practically useful: Probing has been used to investigate LLM representations (Gurnee & Tegmark, 2023; Nanda et al., 2023; Heinzerling & Inui, 2024), de- tect safety relevant quantities (Zou et al., 2023), remove knowledge from models (Elazar et al., 2021), and catch synthetic sleeper agents (MacDiarmid et al., 2024). 2. SAEs might reasonably improve probing: Theoreti- cally, SAE latents are a more interpretable basis of model activations, and we hypothesize that this inductive bias will help train probes in difficult regimes. Recent work has also found some positive results for the utility of SAE probes (Bricken et al., 2024a; Gallifant et al., 2025). Thus, we curate 113 linear probing datasets from a variety of settings and train linear probes on corresponding SAE latent activations (see Figure 2). We compare to a suite of baseline methods across 4 difficult probing regimes: 1) data scarcity, 2) class imbalance, 3) label noise, and 4) co-variate shift. Unfortunately, we find that SAE probes fail to offer a consistent overall advantage when added to a simulated practitioner’s toolkit. Further, we explore areas that Bricken et al. (2024a) suggest SAEs may be valuable for, including detecting attributes distributed over multiple tokens and identifying dataset is- sues. Although SAEs initially seemed promising in these settings, we were able to achieve the same results with im- proved baselines. Our results underscore the necessity for MI works to rigorously design baselines when evaluating the utility of interpretability techniques.Class 0 Activation Vectors SAE Encoder Select top k latents by avg. latent diff across classes i Train logistic regression probe, L1 regularization Class 1 Activation Vectors KNN LogReg XGBoost MLP PCA + LogReg SAE Probe SAE Probe Figure 2. Left: An illustration of our SAE probing method. We pass in training activation vectors from each class and train an L1 regularized logistic regression probe on the latents that differ the most between classes. Right: We ensure robustness of our results with the ”quiver of arrows” approach (see Section 2.4): we add SAE regression into a set of methods, and see if the test accuracy of the best method (chosen by validation accuracy) increases. 2. Methodology We apply probes to the hidden states of two language mod- els, Gemma-2-9B (Riviere et al., 2024) and Llama-3.1-8B (Grattafiori et al., 2024). Our main paper results use Gemma- 2-9B. We replicate core results on Llama-3.1-8B in Ap- pendix F. We use JumpReLU (Rajamanoharan et al., 2024b) sparse autoencoders (SAEs) from Gemma Scope (Lieberum et al., 2024) for Gemma-2-9B and TopK (Gao et al., 2024) SAEs from Llama Scope (He et al., 2024) for Llama-3.1-8B. See Appendix A for further background on SAEs. 2.1. Classification Datasets We collect a diverse set of 113 binary classification datasets listed in Table 3 (Appendix B), including datasets that we expect to be challenging for probes. For exam- ple, 26 headline isfrontpage requires probes to identify front-page headlines and 136 glue mnli entailment requires probes to identify logically entailment. For other example datasets, refer to Table 1. All datasets are titled in the form ID description. We originally collected datasets for other purposes and discarded some of them; thus, while we have 113 datasets, ID ranges from [5, 163]. All datasets contain prompts and targets. Probes are tasked with predicting the target from the model’s hidden activa- tions when run on a prompt. We focus on binary classifi- cation, since SAEs are mostly thought of as representing binarized latents (see Bricken et al. (2023)). Thus, target is either 0 or 1. The prompts in our datasets range in length from 5 tokens to a (left-truncated) maximum of 1024 tokens. 2

Are Sparse Autoencoders Useful? A Case Study in Sparse Probing21 23 25 27 29 Number of Features Used in SAE Probe (k) 0.8 0.9 Average Test AUC Layer 20, Width 16k 100 200 300 400 L0 Figure 3. For a given width, using higher L0 and constructing probes with a larger basis of latents (k) is more performant. of Arrows”: we ask whether adding SAE probes (a new “arrow”) to the set of existing probing methods available to a practitioner (the “quiver”) increases performance compared to a practitioner without access to SAE probes. In other words, over a collection of methods we choose the best method by validation AUC and then report the test AUC of that method. We can then compare the marginal improve- ment of adding SAE probes to a practitioner’s toolkit by comparing the Quiver of Arrows AUC of baeline methods with and without SAE probes. We describe the Quiver of Arrows more formally in Appendix C.4. The quiver of arrows approach is designed as a robustness check to fairly evaluate the benefit of adding SAE probes to a practioner’s toolkit. If we instead constructed probes from a large set of SAEs and chose the best one based on test AUC, we could be tricked into thinking SAEs outperform baselines because we accessed the held-out test set. With the quiver of arrows approach, we emulate the information a real practioner has access to, allowing us to make a robust recommendation on the usefulness of SAE probes. 3. Comparing Probing Techniques in Different Regimes 3.1. Standard Conditions We initially assess probes under standard conditions, char- acterized by sufficient data for probe convergence and bal- anced classes. We find that baseline methods perform the best on layer 20 (see Figure 13a, Appendix D.1), so we run experiments with this layer. We train baselines on 1024 data points (or the maximum number of points in the dataset). We first conduct a preliminary investigation to cut down the large space of Gemma Scope SAEs. We train probes for all SAEs using k latents (for logspaced values of k) on all datasets using 1024 training examples. We calculate the average test AUC for each SAE probe across all datasets. We find that SAE width is relatively unimportant to probe success, while larger L0s lead to more performant probes (Appendix D.2). Additionally, as we might expect, using0.6 0.7 0.8 0.9 1.0 Test AUC (without SAE) 0.6 0.8 1.0 Test AUC (with SAE) Mean : -0.003 ± 0.002 No Change Figure 4. In standard conditions, when SAE probes are added to the quiver, we find a slight decrease in performance. a larger value of k leads to better probe performance, as shown in Figure 3. Thus, throughout the paper we train probes using the largest L0 for SAEs width = 16k, width = 131k, and width = 1M. We use k = 16 to construct easily inter- pretable probes that potentially overfit less and use k = 128 for performance. Using this set of probes, we consider our quiver of arrows approach in standard conditions. SAEs are chosen as the “ar- row” for 14/113 tasks, however, we see a slight decrease in performance when they are added to the quiver in Figure 4. It is unsurprising that SAE probes underperform baselines in standard conditions, as their inductive bias is marginal given a large, balanced dataset. Thus, we now investigate more difficult settings to test if the inductive bias of SAEs translates into a competitive advantage for probing. 3.2. Data Scarcity, Class Imbalance, and Label Noise In this section, we consider more difficult probing regimes: limiting the training data (Data Scarcity), changing the relative frequency of target = 1 examples (Class Imbal- ance), and randomly flipping a fraction of the targets (Label Noise). All of these settings are realistic workflows - for example, a researcher observing a rare model phenomena would like a probe that generalizes with few total exam- ples or few examples of the desired class, while a researcher working with collected user data wants a generalizable probe even if some fraction of users respond randomly. Each regime is characterized by a parameter which we vary to compare SAEs and baselines. • Data Scarcity - We use 20 values of n logspaced in [2, 1024], where n is the number of training examples. We use a standard test set. • Class Imbalance - We use 19 values of ratio linearly spaced between [0.05, 0.95], where ratio = n1 n and n1 is the number of target = 1 training examples. The test set uses the same ratio. • Label Noise - We use 11 values of fraction linearly spaced between [0, 0.5], where fraction = ncorrupted n . We 4

Are Sparse Autoencoders Useful? A Case Study in Sparse Probing105150: periods 116246: spaces 3764: spaces Latent 0 10 20 |Mean Act Difference| (AI - Human) period quest. mark space excl. point Last Token 0 500 1000 1500 Total Activation Human AI Figure 10. Left: Top 3 latents (w/ descriptions) by mean activation difference between AI generated and human SAE latent activations. Right: Histograms of the identity of the 4 most common last tokens in AI generated and human text. 3.5 generated text. Latent 105150 has an AUC of 0.82 on this task, yet appears to fire primarily on periods and punctuation.2 Since this latent is syntactical and unrelated to the content of the dataset, we hypothesize it represents a spurious correlation in the dataset. We examine the final token of each prompt in the dataset and indeed find a spurious feature: AI text is more likely to end in a period while human text is more likely to end in a space (see Figure 10). Although we discovered this by inves- tigating SAE latents, we could reach the same conclusion with baselines: in Table 8 (Appendix G.4), we apply the logistic regression classifier to 2.8 million tokens from the Pile, and find that it is most active on punctuation tokens. 5. Why Didn’t This Work: Illusions of SAE Probes Bricken et al. (2024a) report that SAE probes outperform baselines slightly. A natural question is why our experi- ments fail to replicate this result. One possible explanation is that Bricken et al. (2024a)’s findings were based on a single dataset, whereas we evaluate across multiple datasets. We find that SAE probes outperform baselines on only a small subset of these datasets (2.2%, see Figure 11). It is possible that the dataset used by Bricken et al. (2024a) falls within this subset; however, without access to it, we cannot confirm this. A separate explanation involves a potential illusion due to insufficiently strong baselines. Unlike our work, Bricken et al. (2024a) uses multi-token SAE probing: they aggregate the maximum value for each latent across all prompt tokens, while we only use last token latents. Crucially, they simi- larly max-pool each model dimension across prompt tokens for their comparative baseline, even though activations are unlikely to have privileged dimensions (in Table 9 we show that pooling activations in this way works poorly). We im- plement multi-token SAE probing using max-aggregation 2See Neuronpedia for examples.0 20 40 60 Win Rate Percentage > 0.005 AUC 56.5% 2.2% FAIR Both = Last 39.1% 19.6% ILLUSION Baseline = Last SAE = Pool 34.8% 8.7% FAIR Both = Quiver(Pool, Last) Baseline > SAE SAE > Baseline Figure 11. Illustration of a potentially misleading result we find: when comparing activation probes to pooled SAE probes, SAE probe win rate increases considerably, but adding in a “pooled” attention-inspired probe brings down the SAE win-rate. on 60 random datasets from our list and k = 128; see Ta- ble 9 for full results on these datasets. We also implement a better baseline: we train attention-pooled probes of the form softmaxt∈[1,CtxLen]{Xl t · q} · Xl · v (2) for q, v ∈ Rd. We find that when compared to the last-token baseline, max-pooled SAE probes win 19.6% of the time, a considerable improvement over win rate of last-token SAE probes (2.2%, see Figure 11). However, when implementing attention-pooled baselines and using the quiver of arrows approach to select between pooled and last-token strategies for SAE probes and baselines, the SAE probe win rate drops more than 50% to 8.7% (see Figure 11). 6. Assessing Improvements in SAE Architectures While SAE probes do not robustly outperform baselines, a separate interesting question is whether recent SAE ar- chitectural developments have improved SAE probing per- formance. We investigate this question by examining the performance of eight different SAE architectures released in the last two years, as outlined in Table 2. We use width = 16k Gemma-2-2B layer 12 SAEs with a variety of L0 values trained by Karvonen et al. (2024b). We create k = 16, 128 SAE probes on all regimes and datasets for each architecture, and additionally k = 1 for standard conditions. In Figure 12, we plot each SAE architecture’s k = 16 probe performance in standard conditions. We find the average test AUC of each SAE and then take the max across L0 for each SAE architecture (using a single L0 value is somewhat noisy, see Appendix I). This metric tests how expressive individual SAE latents are for different SAE architectures. While we see a slight positive trend for probing performance with more recently released SAE architectures, the effect is not statistically significant. For plots in all regimes, see Appendix I; in other regimes there 8

Are Sparse Autoencoders Useful? A Case Study in Sparse Probing Table 2. Timeline of SAE Architecture Improvements. SAE Architecture Publication Date ReLU (original) Bricken et al. (2023) October 4, 2023 ReLU (updated) Conerly et al. (2024) April 26, 2024 Gated Rajamanoharan et al. (2024a) May 1, 2024 TopK Gao et al. (2024) June 6, 2024 JumpReLU Rajamanoharan et al. (2024b) July 19, 2024 BatchTopK Bussmann et al. (2024a) July 19, 2024 p-annealing Karvonen et al. July 31, 2024 Matryoshka Bussmann et al. (2024b) December 19, 2024 seems to be a possible slight improvement with newer SAE architectures. 7. Conclusion Our evaluation of sparse autoencoders (SAEs) reveals funda- mental limitations in current SAE methodologies. Despite expectations that interpretable SAE latents would provide a competitive advantage in probing, we found no improve- ment over traditional methods across multiple regimes and over 100 datasets. Some failures expose critical weaknesses – for instance, SAE latents struggle to be robust to distribu- tional shifts and fail to represent complex concepts – while others are more mundane – lower L0 SAEs create worse sparse probes. More broadly, our findings highlight the need for the field of mechanistic interpretability to evaluate tech- niques with rigorous baselines. While prior work reported advantages for SAE probes, our robust quiver of arrows methodology, use of stronger baselines, and evaluation on a large set of datasets demonstrate these advantages to be illusory. Even in our own analysis, initial conclusions favor- ing SAE interpretability were later overturned when proper baselines were considered. We do not consider this work to be a wholesale critique of the SAE paradigm. Instead, we view it as a single rigorously evaluated datapoint to contextualize SAE utility and to motivate a more thorough examination of methods in mechanistic interpretability. Limitations We study the performance of SAE probes as a proxy for SAE utility. While we believe that probing perfor- mance is a more effective measurement than typical SAE evaluations like downstream cross-entropy loss or recon- struction error, it is ultimately still a proxy metric. We chose to study difficult probing regimes to try to find cases whereReLU (orig.) ReLU (new) Gated TopK Jump ReLU Batch TopK p-anneal Matryoshka Architecture (In Order of Release) 0.20 0.15 0.10 0.05 0.00 SAE - Baseline AUC AUC for SAE Architecture vs Baseline k=16 at Best L0 in Standard Conditions Figure 12. We see that there is a slight uptick in probing perfor- mance compared to baselines in standard conditions with more recent architectures. However, the spread of the data is consider- ably larger than this improvement. the inductive bias of SAE latents outweighed imperfect SAE reconstruction, but it is possible that even a basis of “true” model representations would offer only a mild inductive bias advantage over traditional linear probing. We are thus excited for future work studying probing on toy models (e.g. models trained on board games (Karvonen et al., 2024c)) where the “true” model features are known. Finally, it is possible that further optimization of the SAE probe baseline might increase performance such that it beats baseline methods. For example, Gallifant et al. (2025) find that SAE probing with a number of modifications (multi- token probes, binarization of latents, and probing with k = full SAE dimension) beats baseline methods on safety- relevant datasets (albeit only comparing to single-token baseline probes). However, we believe a main takeaway of our paper is that equivalent effort should be put into optimizing baselines. Indeed, we examine the effect of binarizing latents in Appendix J and find that it does not significantly help. 8. Acknowledgments This work was conducted as part of the ML Alignment & Theory Scholars (MATS) Program. We would like to thank the members of the Tegmark group and Neel Nanda’s MATS stream for helpful comments and feedback. JE was supported by the NSF Graduate Research Fellowship (Grant No. 2141064). 9

Are Sparse Autoencoders Useful? A Case Study in Sparse Probing0 20 40 60 Win Rate Percentage > 0.005 AUC 56.5% 4.3% FAIR Both = Last 60.9% 13.0% ILLUSION Baseline = Last SAE = Pool 50.0% 6.5% FAIR Both = Quiver(Pool, Last) Baseline > SAE SAE > Baseline (a) Multi-token non-binarized experiments (the same as Fig- ure 11).0 20 40 60 Win Rate Percentage > 0.005 AUC 56.5% 2.2% FAIR Both = Last 39.1% 19.6% ILLUSION Baseline = Last SAE = Pool 34.8% 8.7% FAIR Both = Quiver(Pool, Last) Baseline > SAE SAE > Baseline (b) Multi-token binzarized experiments. Figure 22. We run the multi-token experiments from Section 5 with and without binarizing aggregated latents. J. Assessing SAE Probe Binarization Gallifant et al. (2025) find that binarizing latents with a threshold improves probe performance. Binarization entails setting a latent equal to 1 if its firing value is greater than a threshold, and setting it equal to 0 otherwise. Gallifant et al. (2025) use a threshold equal to 1. Gallifant et al. (2025) performs probing in the multi-token pooled setting, so to use as similar of a setup as possible we repeat our multi-token experiment with binarization and a threshold equal to 1. Following Gallifant et al. (2025), we binarize after the max-pooling aggregation. As shown in in Figure 22, we find that binarizing results in worse performance than not-binarizing. 27