Step 6: tool results

read_page

{
  "url": "https://arxiv.org/html/2506.23845v2",
  "offset": 3000,
  "max_chars": 20000
}

{"ok":true,"url":"https://arxiv.org/html/2506.23845v2","text":"s of SAEs. For example, in response to this research, the mechanistic interpretability team at Google DeepMind announced that they would deprioritize research into SAEs [ Smith et al., 2025 ] . Nonetheless, there continues to be optimism about new applications of SAEs, including in hypothesis generation [ Movva et al., 2025b ] and in the “biology” of LLMs [ Lindsey et al., 2025 ] .\nHow can we square continued interest in SAEs with thorough evaluations demonstrating negative results? Are new attempts to use SAEs misguided? Or is there something missing in our understanding of the negative results? This position paper reconciles conflicting narratives surrounding SAEs by making a conceptual distinction. Our position is that SAEs—even if less effective for acting on known concepts —are powerful tools for discovering unknown concepts .\nConsider the tasks where negative results have been shown. Concept detection involves detecting a known, prespecified concept (“Does this text mention dogs?”). Model steering involves steering a model to exhibit a specified concept (“Make outputs less sycophantic.”).\nAnother negative result involves concept unlearning [ Farrell et al., 2024 ] (“Unlearn knowledge about concepts related to biosecurity.”).\nIn these tasks, concepts are inputs —known beforehand.\nNow consider the tasks where positive results have been shown. Hypothesis generation involves finding concepts that predict a target variable (“What concepts predict engagement of news headlines?”). Biology of LLMs involves finding concepts that LLMs represent when generating text (“What concepts does an LLM represent when doing addition?”). In both tasks, concepts are outputs ---unknown beforehand.\n1 1\n1\nWhen we say “unknown” concept, we mean that the researcher or practitioner cannot specify the particular concept that answers their question. This concept (once revealed to the researcher) may be well understood. For example, a researcher may be very familiar with the concept of “surprise,” but did not know beforehand that this concept predicted an outcome of interest (e.g., number of clicks on a headlines). Furthermore, the concept also may be stored in language model representations; therefore, the primary task is to uncover the right concepts—a task we argue SAEs are well-equipped to do.\nSo even given results showing that SAEs underperform baselines when acting on known concepts, SAEs remain promising and underexplored tools for discovering unknown concepts. By enumerating concepts in an unsupervised manner, SAEs allow for the discovery of concepts that fit desired criteria. We outline how SAEs—as a tool for generating unknown concepts—can be used to advance research in (i) ML interpretability, explainability, fairness, auditing, and safety, and (ii) discovery in the social and health sciences. For example, in ML fairness and auditing, researchers can use SAEs to discover previously unknown concepts that bias model outputs. In the health sciences, researchers can use SAEs to discover previously unknown concepts that predict health outcomes, or to discover spurious correlations in existing predictive models.\n(On the other hand, we do not intend to suggest that SAEs do not also have potential in the known concepts regime. There are increasingly positive results in this setting compared to baselines [ Cywiński and Deja, 2025 , Arad et al., 2025 ] , especially when considering cost and efficiency [ Nguyen et al., 2025 ] . Our paper focuses on establishing how SAEs are particularly useful in the unknown concepts regime, and that negative results in the known concepts regime need not dissuade researchers from exploring the use of SAEs.)\nPaper structure.\nSection 2 serves as a primer on SAEs (which can be readily skipped by readers familiar with SAEs). Section 3 shows that negative SAE results pertain to tasks that act on known concepts. Section 4 surveys recent positive results, showing that these papers use SAEs to discover unknown concepts. Section 5 then explores use cases for SAEs in different research areas.\n2 An SAE Primer\nWe offer a brief primer on the SAE architecture, their history, and why and how they are now being used to interpret language models. (Readers familiar with SAEs may skip to Section 3 .)\nEarly work on autoencoders.\nAutoencoders are unsupervised neural networks that learn to reconstruct high-dimensional inputs via a series of learned transformations.\nFor a D D -dimensional input 𝐱 \\mathbf{x} , an autoencoder computes:\n𝐳 = encoder ⁡ ( 𝐱 ) , \\displaystyle\\mathbf{z}=\\encoder(\\mathbf{x}),\n(1)\n𝐱 ^ = decoder ⁡ ( 𝐳 ) , \\displaystyle\\hat{\\mathbf{x}}=\\decoder(\\mathbf{z}),\n(2)\nwhere encoder ⁡ ( ⋅ ) , decoder ⁡ ( ⋅ ) \\encoder(\\cdot),\\decoder(\\cdot) are arbitrary neural networks, 𝐳 \\mathbf{z} is the latent feature representation, and 𝐱 ^ \\hat{\\mathbf{x}} is the reconstruction . The autoencoder is trained with a mean squared error reconstruction loss,\nℒ = ‖ 𝐱 ^ − 𝐱 ‖ 2 2 . \\displaystyle\\mathcal{L}=||\\hat{\\mathbf{x}}-\\mathbf{x}||_{2}^{2}.\n(3)\nOne classic application of autoencoders is compression: by restricting the latent representation 𝐳 \\mathbf{z} to a dimension size M ≪ D M\\ll D , the autoencoder learns a compressed representation in 𝐳 \\mathbf{z} which can be used to approximate 𝐱 \\mathbf{x} [ Hinton and Salakhutdinov, 2006 ] .\nIn this setting, 𝐳 \\mathbf{z} functions similarly to an M M -dimensional principal component analysis of 𝐱 \\mathbf{x} , 2 2\n2\nIf the encoder and decoder are linear, PCA minimizes the reconstruction loss [ Baldi and Hornik, 1989 ] . in that we wish to explain as much variance as possible in the distribution of 𝐱 \\mathbf{x} using only M M dimensions.\nSparse autoencoders.\nSparse autoencoders (SAEs) perform the same reconstruction task, but leverage a different intuition.\nIn an SAE, M M can be larger than D D , but each individual 𝐳 \\mathbf{z} is forced to be sparse—that is, only a small number of its dimensions can be nonzero.\nThis design is motivated by the idea that while an entire dataset (e.g., all text on the Internet, or all images in ImageNet) may span many possible concepts, a single datapoint (a sentence or image) often contains very few.\nEmpirically, enforcing this structure often produces latents in 𝐳 \\mathbf{z} that capture atomic concepts.\nFor example, early work on SAEs trains directly on input images 𝐱 \\mathbf{x} (e.g., from MNIST), and 𝐳 \\mathbf{z} contains interpretable features like edges [ Coates and Ng, 2011 , Makhzani and Frey, 2014 ] .\nTo improve clarity, we define features and concepts :\n•\nA feature is one of many numerical values used to represent an input. In a neural network, a feature is a single dimension of a layer’s output vector.\nIn an SAE specifically, a feature is an activation computed by a single latent neuron .\nWe use the terms feature, activation, neuron, and dimension interchangeably depending on context.\n•\nA concept is a qualitative characteristic that may or may not be present in a given input. Here, we operationalize concepts via natural language descriptions.\n•\nAn interpretable feature , then, is a feature whose values correspond to the presence or absence of a single concept.\nTable 1 : SAE latent features mapped to natural language concepts via autointerpretation, and texts that activate them [ Movva et al., 2025b , Lindsey et al., 2025 ] . Left: Examples of concepts learned from SAEs trained on different datasets; Right: Examples of texts that activate the corresponding SAE feature. Concepts interpretably describe the underlying data distribution of texts.\nMathematical formulation of SAEs.\nOne formulation of the sparse autoencoder follows the usual autoencoder forward pass, but adds an L 1 L_{1} penalty on 𝐳 \\mathbf{z} to the loss function [ Coates and Ng, 2011 ] :\nℒ = ‖ 𝐱 ^ − 𝐱 ‖ 2 2 + λ ​ ‖ 𝐳 ‖ 1 . \\displaystyle\\mathcal{L}=||\\mathbf{\\hat{x}}-\\mathbf{x}||_{2}^{2}+\\lambda||\\mathbf{z}||_{1}.\n(4)\nA larger λ \\lambda encourages more zero elements in 𝐳 \\mathbf{z} .\nAnother approach removes the L 1 L_{1} penalty, but explicitly applies a TopK \\topk function to the encoder that zeroes out all but the K K largest activations in 𝐳 \\mathbf{z} [ Makhzani and Frey, 2014 ] , where K ≪ M K\\ll M .\nThe full forward pass is given by:\n𝐳 \\displaystyle\\mathbf{z}\n= ReLU ⁡ ( TopK ⁡ ( W enc ​ ( 𝐱 − 𝐛 pre ) + 𝐛 enc ) ) , \\displaystyle=\\relu(\\topk(W_{\\enc}(\\mathbf{x}-\\mathbf{b}_{\\mathrm{pre}})+\\mathbf{b}_{\\enc})),\n𝐱 ^ \\displaystyle\\hat{\\mathbf{x}}\n= W dec ​ 𝐳 + 𝐛 dec , \\displaystyle=W_{\\dec}\\mathbf{z}+\\mathbf{b}_{\\dec},\nwhere 𝐛 pre ∈ ℝ D , W enc ∈ ℝ M × D , 𝐛 enc ∈ ℝ M , W dec ∈ ℝ D × M , 𝐛 dec ∈ ℝ D \\mathbf{b}_{\\mathrm{pre}}\\in\\mathbb{R}^{D},W_{\\enc}\\in\\mathbb{R}^{M\\times D},\\mathbf{b}_{\\enc}\\in\\mathbb{R}^{M},W_{\\dec}\\in\\mathbb{R}^{D\\times M},\\mathbf{b}_{\\dec}\\in\\mathbb{R}^{D} .\nTop- K K SAEs with a single-layer encoder and decoder (as above) have emerged as a common architecture in recent work, with slight variations to mitigate issues like dead neurons and feature absorption [ Gao et al., 2024 , Bussmann et al., 2025 ] .\nSome work has replaced SAEs with sparse transcoders , which use layer ℓ i \\ell_{i} to construct the output of a later layer ℓ j \\ell_{j} [ Paulo et al., 2025 , Lindsey et al., 2025 ] .\nFor convenience, we refer to all of these closely-related sparse coding methods under the “SAE” umbrella , while noting that the specific optimal architecture is likely to shift.\nApplying SAEs to interpret language models.\nThe recent wave of SAE research aims to interpret the representations learned by large language models.\nThe motivation for this line of work is to understand the units and computations an LM uses to map inputs to outputs.\nBefore SAEs, a plethora of works over the last decade on probing language models have shown that LM token representations contain rich semantic information [ Belinkov, 2022 ] .\nConcepts like a word’s part-of-speech or pronoun co-references are a linear transformation away from the word’s representation [ Liu et al., 2019 ] .\nGiven this richness, a natural question is whether we can identify all of the concepts that a language model encodes.\nUnfortunately, neurons in a language model tend to be polysemantic : they encode many concepts at once [ Elhage et al., 2022 ] , so it is difficult to extract an LM’s concepts by studying its neurons.\nThis convergence of findings—that language model representations encode numerous valuable concepts, but studying individual neurons does not reveal them—explains recent excitement for sparse autoencoders.\nUnlike LM neurons, SAEs produce monosemantic neurons that can be explained by a single concept [ Cunningham et al., 2023 , Bricken et al., 2023 ] .\nSAEs are trained on an LM’s representations 𝐱 \\mathbf{x} of individual tokens, resulting in latent representations 𝐳 \\mathbf{z} .\nTo interpret a particular feature dimension i i in 𝐳 \\mathbf{z} , we can examine tokens that produce large values of 𝐳 ⁡ [ i ] \\mathbf{z}[i] .\nInitial work reports that after training on the representations from a small one-layer LM, the SAE features have succinct meanings like “ Arabic text ” or “ citations in scientific papers ” [ Cunningham et al., 2023 , Bricken et al., 2023 ] .\nFollow-up work demonstrates that SAEs scale to state-of-the-art LLMs [ Templeton et al., 2024 , Gao et al., 2024 ] , and that they can be used to interpret embeddings of text chunks rather than just individual tokens [ O’Neill et al., 2024 ] .\nIn Table 1 , we provide examples of concepts learned on both specific text datasets (news headlines and Congressional speeches) as well as general text corpora. SAEs may also be applied to other models, such as vision or biological models\n[ Simon and Zou, 2025 , Brixi et al., 2025 , Adams et al., 2025 ] .\nAutomatically interpreting features with language models.\nWhile SAEs produce features in 𝐳 \\mathbf{z} that are theoretically interpretable, generating a mapping from features to concepts is a separate challenge.\nThere are too many features to explain manually, so prior work has focused on automatically generating explanations 3 3\n3\nMany key works on automatic explanation interpret neurons in language or vision models directly, without SAEs [ Bau et al., 2017 , Hernandez et al., 2022 , Bills et al., 2023 , Choi et al., 2024 ] . The value proposition of SAEs is that, compared to the original neurons, SAE features can be explained with much higher fidelity. ( Templeton et al. [2024] , O’Neill et al. [2024] , inter alia ).\nTo interpret a feature i i , a basic approach is to prompt a language model with texts that have a high value of 𝐳 ⁡ [ i ] \\mathbf{z}[i] against those with a low value, and ask it to identify the shared concept in the high-valued texts.\nTo evaluate the quality of the resulting concept description, one can use an LM to annotate a subset of texts for the presence of the concept, and measure agreement between the concept annotations and the true feature values.\nThis framework yields a quantitative score for interpretability, measuring how well a feature can be explained in natural language.\nOther concept-based architectures.\nIt is helpful to contrast the capabilities of sparse autoencoders with seemingly-similar methods like concept bottleneck models [ Koh et al., 2020 ] . While both have neurons that fire in the presence of concepts, in concept bottleneck models these concepts are prespecified before training the model 4 4\n4\nIn the original formulation, concepts are chosen by domain experts [ Koh et al., 2020 ] ; recent work uses language models to propose concepts [ Sun et al., 2025 , Ludan et al., 2024 ] . In both cases, the concepts need to be known beforehand, either by an expert or an LLM. , while in sparse autoencoders, the concepts are learned while training the model. In other words, in the case of SAEs, researchers need not know what concepts are useful beforehand.\nNeg. Results: Acting on Known Concepts\nPos. Results: Discovering Unknown Concepts\nConcept detection [ Wu et al., 2025 , Kantamneni et al., 2025 ]\nHypothesis Generation [ Movva et al., 2025b ]\nIs the following name a basketball player?\nWhat concepts predict engagement on news headlines?\nIs the following entity in New York City?\nWhat concepts predict partisanship in Congressional speeches?\nModel steering [ Wu et al., 2025 ]\nBiology of LLMs [ Lindsey et al., 2025 ]\nMake the LLM output more sycophantic.\nWhat concepts does an LLM represent after writing the first line of a poem?\nMake the LLM output discuss the Golden Gate Bridge.\nWhat concepts does an LLM represent when performing addition?\nTable 2 : Negative SAE results act on known concepts whereas positive SAE results focus on discovering unknown concepts .\n3 Negative results: Acting on Known Concepts\nWe now survey recent negative results about SAEs, with the goal of showing that the tasks considered fall under the category of acting on known concepts. This is to be contrasted with tasks that involve discovering unknown concepts, on which positive results have been shown (Section 4 ).\nTwo recent papers conduct large-scale evaluations of SAEs [ Kantamneni et al., 2025 , Wu et al., 2025 ] . A key finding of these papers is that SAEs underperform simple baseline methods (such as logistic regression or naive prompting). We claim that these evaluations are limited to tasks involving acting on known concepts . Indeed, the tasks that are studied are:\n1.\nConcept detection [ Kantamneni et al., 2025 , Wu et al., 2025 ] : Identifying whether a given concept appears in a text.\n2.\nModel steering [ Wu et al., 2025 ] : Steering the outputs of a language model to contain a concept.\nThese are important, widely-studied problems, and understanding how SAEs perform on them is clarifying. Notice, however, that these tasks each involve first prespecifying a concept and then acting upon it. In other words, in these tasks, concepts are inputs. We now summarize these papers’ findings in greater detail.\nConcept detection.\nKantamneni et al. [2025] curate 113 binary classification tasks on text data, which they use to evaluate concept detection accuracy.\nFor example, one task is to determine whether a given name corresponds to a basketball player.\nAnother task is to determine whether a tweet conveys happy sentiment.\nFor each such task, they fit a classifier to predict the presence of the concept using Gemma-2-9B ’s representations of the final token in each text as input.\nThey compare this to a classifier trained on the latent representations from a Gemma-2-9B SAE.\nThey further examine class imbalance, data scarcity, and label noise.\nIn each setting, the classifier trained on top of the SAE does not outperform the classifier trained directly from the LM.\nWu et al. [2025] follow a similar approach.\nStarting from a list of 500 concepts, for each concept, they generate synthetic texts that either do or do not contain the concept.\nIn addition to logistic regression using Gemma-2-2B representations, they train several other representation-based concept detection methods.\nThey also include methods that do not use representations at all, such as prompting an LLM to identify whether the concept is present in the text, as well as bag-of-words.\nFour such baselines, including logistic regression and prompting, outperform the SAE.\nModel steering.\nWu et al. [2025] also study model steering.\nGiven a user prompt and a concept, like “where should I visit today?” and “Golden Gate Bridge,” they evaluate whether the model can generate a response that is fluent, relates to the prompt, and includes the concept.\nAn LLM judge scores each attribute.\nTo steer with an SAE, they identify the SAE feature that is most predictive of the concept’s presence, and they generate a response after increasing the value of this feature.\nNon-SAE methods include editing activations with a steering vector [ Marks and Tegmark, 2024 ] , finetuning on responses containing the concept, or simply prompting to include the concept in the response.\nPrompting and finetuning both outperform SAE-based steering.\nWhat explains these negative results?\nWe suggest some reasons why SAEs underperform baselines on these tasks.\nFor concept detection, recall that SAEs are trained to reconstruct the LM token representations.\nA reconstruction encodes strictly less information about a token than the original LM representation.\nIt follows that, compared to the original representation, there is less information available in the SAE representation to predict the presence of a concept.\nFor model steering, prompting performs well because LLMs are finetuned to be adept at instruction-following, and including a concept in a response falls well within this paradigm.\nThe empirical results from both papers underscore an intuition that there are many natural methods besides SAEs to act on known concepts. (Though, methodological innovations may make SAEs more competitive at these tasks as well [ Arad et al., 2025 ] .) Crucially, however, these baselines are less equipped to perform another simple task: to enumerate a list of unknown concepts that satisfy some objective. This, as we show in the next section, forms the basis for tasks on which SAEs have a comparative advantage.\n4 Positive results: Discovering Unknown Concepts\nWe now describe two positive results using SAEs 5 5\n5\nNote that Lindsey et al. [2025] use sparse coding methods that differ slightly from SAEs (see note at end of § 2 ). [ Movva et al., 2025b , Lindsey et al., 2025 ] , which focus on the following tasks:\n1.\nHypothesis generation [ Movva et al., 2025b ] : Identifying open-ended natural language concepts that predict a target variable.\n2.\nExplaining language model outputs (“Biology of LLMs”) [ Lindsey et al., 2025 ] : Describing the concepts a language model uses to perform various tasks (e.g., poem completion or addition).\nWe claim that these tasks are examples of discovering unknown concepts . To explain this, we summarize their findings in greater detail.\nHypothesis generation.\nMovv","chars":20000,"total_chars":53077,"offset":3000,"truncated":true,"next_offset":23000}