Step 4: tool results

provenance_read_source

{
  "max_chars": 20000,
  "source_id": "ff9222f2-759d-4782-a7ff-8c811f759780"
}

{"success":true,"source":{"id":"ff9222f2-759d-4782-a7ff-8c811f759780","url":"https://arxiv.org/pdf/2605.31183","title":"Steering LLMs? Actually, Sparse Autoencoders can outperform simple baselines","source_type":"preprint"},"origin":"stored","total_chars":55411,"offset":0,"content":"Steering LLMs? Actually, Sparse Autoencoders can outper-\nform simple baselines\nMikkel Godsk Jørgensen mikkel.godsk.research@gmail.com\nDTU Compute\nTechnical University of Denmark\nLars Kai Hansen lkai@dtu.dk\nDTU Compute\nTechnical University of Denmark\nAbstract\nSparse Autoencoders (SAEs) have been seen as a promising avenue for exploring the inter-\nnals of Large Language Models (LLMs) and for steering model output generation. When\nAxBench - a model steering benchmark - was introduced in Wu et al. (2025), SAEs did not\nseem to live up to their original hype due to poor steering performance relative to a set of\nsimple baselines. This work serves as a partial rebuttal for Sparse Autoencoders and sug-\ngests that the results of Wu et al. (2025) did not do them full justice. We find that Sparse\nAutoencoders can, in fact, perform close to on par with the reference LoRA performance\non the AxBench benchmark, when features are selected and labelled with our supervised\npipeline. We also find that our pipeline selects features that are surprisingly causal of their\nidentified labels when using only its interpretability-based components. Lastly, we present\nevidence that high sparsity (low ℓ0) may not be crucial for successful steering based on\ninterpretability, which is in contrast to the earlier findings in Wang et al. (2025). Our code\nis available at https://github.com/MikkelGodsk/SAE-labelling.\n1 Introduction\nSparse Autoencoders have recently attracted a wave of attention (e.g. Cunningham et al., 2023; Makelov\net al., 2024; Templeton et al., 2024; Gao et al., 2024; Rajamanoharan et al., 2024), offering an unsupervised\ninterpretability model based on the Linear Representation Hypothesis (LRH) and the superposition hypoth-\nesis. The LRH suggests that many concepts may be stored in linear subspaces, as discussed in e.g. Alain\n& Bengio (2018); Kim et al. (2018); Elhage et al. (2022); Park et al. (2023), although somewhat contested\nin e.g. Crabbé & van der Schaar (2022). Since LLMs can represent a large number of concepts in relatively\nlow-dimensional latent spaces, the neurons must be polysemantic as a consequence, i.e. they become ac-\ntive across multiple different semantics, as discussed in e.g. Bolukbasi et al. (2021); Makelov et al. (2024);\nTempleton et al. (2024); Fereidouni et al. (2025); Cunningham et al. (2023). The superposition hypothesis\nfurther suggests that linear concept representations are densely packed and not fully orthogonal (Elhage\net al., 2022). Within this framework, Sparse Autoencoders offer an appealing interpretability tool, capable\nof discovering a subset of linear concept representations, referred to as features, which are thought to exhibit\na high degree of monosemanticity (e.g. Bricken et al., 2023; Templeton et al., 2024). In Fereidouni et al.\n(2025), the authors quantify the monosemantic property of SAEs in comparison to individual neurons in\na network. Their findings suggest that, indeed, SAE features are more monosemantic than the individual\nneurons of language models.\nWhile the monosemantic response is interesting for interpretability, features have also shown some ability\nto control the model’s text generation (e.g. Templeton et al., 2024; Arad et al., 2025; He et al., 2025; Wang\net al., 2025), although the effectiveness is disputed most notably in Wu et al. (2025). In fact, Wu et al.\n(2025) concludes that “Even simple baselines outperform Sparse Autoencoders” for steering, and finds that\n1\narXiv:2605.31183v1 [cs.CL] 29 May 2026\n\nsimply prompting the model is much more effective than other approaches, such as intervention via SAEs\n(e.g. Templeton et al., 2024) and LoRA (Hu et al., 2022). Although this is undeniably important for most\npractical applications, the prompt baseline can seem unfair since the subject LLM was originally trained\nprecisely to follow instructions. Aside from steering, there have also been several attempts at using SAEs for\nknowledge unlearning/concept removal in language models (e.g. Farrell et al., 2024; Yamashita et al., 2025;\nFereidouni et al., 2025), essentially exploring the negative (removal) dimension of steering.\nSince Sparse Autoencoders are trained unsupervised, efforts have been made to label their features with the\nconcepts they represent. This has been attempted at scale using LLMs (e.g. Lieberum et al., 2024; Paulo\net al., 2024), but to our knowledge, no dedicated efforts have been made to use a supervised approach to\nlabelling.\nRecent work (Wang et al., 2025) has suggested that high interpretability of a feature tends to have some corre-\nlation with high utility for model steering, and that this relationship seems most pronounced for JumpReLU\nSAEs with a low ℓ0 (i.e. high sparsity). This result could pose as a relevant context for interpreting the\nfindings of Wu et al. (2025), in which they base their conclusions on SAEs with high ℓ0, being those that are\nalready annotated in Neuronpedia1.\n1.1 Contributions\nWe make the following contributions:\n1. We introduce a labelling pipeline for Sparse Autoencoders based on labelled datasets rather than\nLLMs.\n2. We demonstrate that Sparse Autoencoders are substantially more powerful for steering than re-\nported in Wu et al. (2025) when the features are labelled via our proposed method. However, the\nsteering performance we demonstrate is still not fully competitive with prompting on the AxBench\nbenchmark.\n3. We find indications that both low- and high-sparsity Sparse Autoencoders in Gemma Scope can be\nused for steering.\n2 Preliminaries\n2.1 Large Language Models\nFormal definition. Large Language Models define a distribution over text strings, and are typically based\non transformers (Vaswani et al., 2023). To transform a string of text into a format suitable for language\nmodeling, the typical approach is to use a tokenizer to split up the text into a string of tokens. Here, the\ntokens constitute an alphabet Σ. For modeling purposes, we further augment the alphabet by the tokens\nBOS and EOS (beginning of sequence, end of sequence) to get an augmented alphabet ¯Σ = Σ ∪ {BOS, EOS}.\nFormally, a sequence of tokens is simply called a string. The language model can then be characterized as a\nstring distribution\npLM(s) = ∏\ni\npSM(si|s<i), s ∈ (BOS · Σ∗ · EOS) , (1)\nwhere · denotes string concatenation and pSM(·|s<i) : ¯Σ → [0, 1] is typically fitted by the model. Here pLM\nis assumed to be a true distribution (i.e., no probability mass is leaked into the set of infinite strings). This\nis true for models with a softmax-based prediction head. To use the language model generatively, we use a\ndecoding strategy to auto-regressively predict the next token until termination by EOS. The two simplest\noptions are to greedily select arg maxsi pSM(si|s<i) as our next token, or to sample from pSM(si|s<i) at each\nstep (Cotterell et al., 2024).\nTransformers. LLMs typically build on a modified transformer architecture compared to the one introduced\nin Vaswani et al. (2023). The main modifications are to use only the decoder, to apply the positional\n1https://www.neuronpedia.org/gemma-scope#browse\n2\n\nFigure 1: The architecture of a transformer-based causal LM at the level of a single transformer block. Here\nwe see the residual stream as the vertical line running through the block, and the consequences of the causal\nmask on the attention. For simplicity, we assume the example words are mapped to single tokens and leave\nout the embedding and decoding step, as well as the positional encoding. The model generation bracket\nshows the autoregressive nature of the model.\nencodings within the Multi-Head Self-Attention, and to use the causal mask. The causal mask has the effect\nof making information always flow forward over a sequence of tokens. E.g., for two sentences \"The bar closes\nat 4\" and \"The bar is made of metal\", the internal representation for the word \"bar\" cannot be disambiguated\nat its token position using the succeeding tokens as they are masked out. This has the advantage of being\ncomputationally cheaper since the current representations need not be recomputed every time a new token\nis generated. This is depicted in figure 1. For the reader less familiar with LLMs, we will occasionally refer\nto the chain of residual connections that flows throughout the network as the \"residual stream\". This view\nhas been widely adopted in the literature, aptly so since there is a direct path from the input layer to the\noutput layer through it, where the transformer blocks can be seen as reading and writing its value (Elhage\net al., 2021; Lieberum et al., 2024).\nPrompt Steering. Prompting offers a straightforward way to introduce a concept into a model generation.\nFor instance, if we want the language model to generate a poem about baseball, a natural approach is to\nuse the prompt: \"Write a poem about baseball\". In this work, we follow the procedure of Wu et al. (2025).\nHere we ask GPT-4o-mini to write a prompt explaining our subject LLM to include a topic (e.g., \"baseball\")\ninto its generation, to which we append the instruction (e.g., \"Write a poem\"). One such prompt could have\nthe format:\nYou are a language model that must always incorporate the concept of baseball into your\nresponses, [...]\nWrite a poem.\n2.2 Sparse Autoencoders\nSparse Autoencoders are attributed to Ng et al. (2011). A popular interpretation of SAEs is that they learn\nan over-complete dictionary of features. An alternative view is as a set of linear probes without known labels,\nwhich is the view we adopt.\n3\n\nDefinition. SAEs are shallow, wide autoencoders, typically trained to reconstruct activations while pro-\nmoting sparsity in the latent layer. Formally, the model can be described as\n(Encoder): z(x) = σ(xWenc + benc) (2)\n(Decoder): ˆx(z) = zWdec + bdec, (3)\nwhere x ∈ Rdmodel , ˆx ∈ Rdmodel , and z ∈ RdSAE respectively denote the network activation, the activation\nreconstruction and the feature activation vector. In SAEs, we have dmodel ≪ dSAE. We can consider the\nencoder as discriminative/detective, and the decoder part as generative. As demonstrated in Haufe et al.\n(2014), the discriminative component may not exclusively reflect our signal of interest, hence we do not\nexpect the columns of Wenc to be clean feature representations. Instead, due to the generative nature of the\ndecoder, it is more natural to use the rows of Wdec as the learned feature representations in concordance\nwith e.g. Bricken et al. (2023); Templeton et al. (2024). It is important to note that these features are not\nguaranteed to be an exhaustive list (Paulo & Belrose, 2026). In the case of Gemma Scope (Lieberum et al.,\n2024), the activation σ(·) is the JumpReLUθ function (originally introduced in Erichson et al., 2019):\nσ(v) = JumpReLUθ (v) = v ⊙ 1 [v ≥ θ] (4)\nwhere ⊙ is the element-wise (Hadamard) product. The encoder has learnable weights Wenc ∈ Rdmodel×dSAE ,\nlearnable bias benc ∈ RdSAE , and a learnable threshold θ ∈ RdSAE . The decoder has the learnable weights\nWdec ∈ RdSAE×dmodel , and the learnable bias bdec ∈ Rdmodel . Together, the model is trained to minimize\nL = ||x − ˆx||2\n2 + λ||z(x)||0, where the second term incentivizes sparsity on feature activations (Lieberum\net al., 2024; Bricken et al., 2023; Gao et al., 2024; Rajamanoharan et al., 2024). For a technical explanation\non how this is performed in practice, refer specifically to Rajamanoharan et al. (2024).\nFeature Steering. In this work, we use the approach of Templeton et al. (2024) for activation editing.\nDuring inference, we clamp the activation of a feature (z(x))f to a value α. The specific approach first\ncomputes the reconstruction error term\ne = x − ˆx(z(x)), (5)\nfollowed by the latent vector after intervention via editing;\nzint = z(x) ⊙ (1 − m) + αm (6)\nwhere m is a mask consisting of elements mj = 1 [j = f ] . The edited activations can finally be computed as\nˆxint = ˆx(zint) + e, (7)\nwhich we pass through the rest of the model. This is done on all tokens, including the BOS token. We shall\nrefer to this procedure as feature steering.\n3 Method\n3.1 Labelling pipeline\nTo label the learned features of an SAE, we propose to construct linear probes based on single features\nand match their predictions with labels on a multi-label dataset. Here we use the LLM activations X ∈\nR|D|×T ×dmodel computed over a dataset D with maximum sequence length T , as well as a binary mask\nM ∈ {0, 1}|D|×T zeroing out token positions not included in a given sample. The dataset comes with a\nbinary label matrix Y ∈ {0, 1}|D|×nlabels .\nFeature probes. We construct the probe predictions by first computing the feature activations Zi,j,: =\nz (Xi,j,:) resulting in a sparse tensor Z ∈ R|D|×T ×dSAE . Given the activations of feature f , we then compute\nits activation frequency on the i’th text example:\nFi,f =\n\n\nT∑\nj=1\nMi,j\n\n\n−1 T∑\nj=1\nMi,j · 1 [Zi,j,f > 0] ∈ [0, 1], (8)\n4\n\nusing the indicator function 1[·]. We define ˆyi,f,τ = 1 [Fi,f > τ ] to form the probe prediction given a threshold\nτ . Since it is not obvious what would constitute a good frequency threshold, we let τ be a free parameter.\nProbe labelling. To match a dataset label to a probe, we compute a match score using the calibrated\nF1 (Siblini et al., 2020) followed by additional denoising. For every feature-label pair (ℓ, f ), we define the\ncalibrated F1 score as:\nTPℓ,f,τ =\n|D|\n∑\nj=1\n1 [yj,ℓ = 1 ∧ ˆyj,f,τ = 1] , FPℓ,f,τ =\n|D|\n∑\nj=1\n1 [yj,ℓ = 0 ∧ ˆyj,f,τ = 1] , (9)\nFNℓ,f,τ =\n|D|\n∑\nj=1\n1 [yj,ℓ = 1 ∧ ˆyj,f,τ = 0] , πℓ = 1\n|D|\n|D|\n∑\nj=1\n1 [yj,ℓ = 1] , (10)\nF1c\nℓ,f,π0 := max\nτ\n\n 2TPℓ,f,τ\n2TPℓ,f,τ + (1−π0)πℓ\nπ0(1−πℓ) FPℓ,f,τ + FNℓ,f,τ\n\n . (11)\nWe further explore this definition in appendix A and examine the effects of the parameter π0. In practice,\nthe max operation selects the optimal threshold τ over a set of candidates {0.05, 0.1, ..., 0.95}.\nTo motivate the use of the F1 score, consider its decomposition into the harmonic mean between precision\nand recall, i.e. F1 = 2 (precision−1 + recall−1)−1 . The objective is for a probe to have both high precision\n(almost exclusive activation) and high recall (activation on most occurrences) for the correct label. A probe\nscoring highly in one but low in the other would likely correspond to a super- or a sub-category, or something\nsemantically adjacent. We here took inspiration from insights of Gao et al. (2024); Gurnee et al. (2023). To\nmotivate the calibration, we consider how the ordinary F1 score is sensitive to the label support (the number\nof occurrences for the label). Since labels may have different supports, the scale of the F1 score will vary\nbetween labels, rendering it unreliable for comparing the degree of agreement between a feature and a label.\nAgain, we refer the reader to appendix A for a detailed exploration.\nTo select a set of valid matches based on F1c\nℓ,f,π0 , we devise a set of criteria to denoise the result. These are\nperformed in the following order:\n1. To reduce variance, we threshold the label support, i.e., the number of label occurrences in the\ndataset. We found 50 to be a good threshold through parameter tuning, which we elaborate in\nsection 4.\n2. We use the output score from Arad et al. (2025) as a proxy for the steering ability of a feature. We\nuse a threshold of 10−3 found by visually inspecting the output score distribution and choosing it\nto only include a small best subset.\n3. We keep the top-K of the remaining feature-label pairs. We use K = 50, since it provides a tradeoff\nbetween accuracy and number of matches.\n3.2 AxBench\nAxBench is a benchmark for model generation under steering, prompting the model with Alpaca-Eval in-\nstructions (Wu et al., 2025; Li et al., 2023). The generated texts are rated by GPT-4o with scores {0, 1, 2}\non 1) fluency (whether they are correct English); 2) instruction following (whether they respond to the given\ninstruction); and 3) concept incorporation (whether they naturally include the target concept). The scores\nare finally combined into a single overall score for each generation by the harmonic mean\naggregated_rating = 3\ninstruction_rating−1 + concept_rating−1 + fluency_rating−1 .\nThe ordinary mean is used to aggregate each score over the dataset.\nFeature Steering Evaluation. To generate a set of steered texts, we first draw 20 random instructions\nfrom Alpaca-Eval without replacement. We repeat this for each suggested feature-label pair. For a set of\n5\n\nconsidered steering strengths α ∈ {100, 125, ..., 575}, we generate feature-steered responses to the selected\ninstructions using each strength value.\nIn Wu et al. (2025), they select α by maximizing aggregated_rating in holdout (5 training examples, 5 test\nexamples). The resulting value is used across all features. However, since we do not expect all features\nto require the same α for optimal steering, we select it individually for each feature-label pair. Here, we\ngenerate 20 responses for every considered α, for every suggested feature-label pair. We then report the\nresult from repeated cross-validation (10 repeats, 4 splits) to reduce variance2. To evaluate the overall\nsteering, we use aggregated_rating as the maximization objective, as in Wu et al. (2025). However, since the\naggregated rating accounts for model-specific abilities, we use the concept_rating to evaluate the accuracy\nof our feature-labels.\nTo offer an additional motivation for a steering-based pipeline evaluation, this approach avoids the potential\nissue of dataset idiosyncrasy that may cause an interpretability illusion, as observed in Bolukbasi et al. (2021).\nHere, the observed semantics of a neuron vary across the datasets used to identify them, possibly explained\nby neuron polysemanticity. In contrast, feature steering is a causal experiment, and while it additionally\ndemands causation, a successful feature steering would confirm what we may consider the \"ground truth\"\nlabels of the learned features.\nPrompt Steering Evaluation. For each feature-label pair, we use the same 20 random instructions. We\nthen generate the prompt-steered responses to each instruction. Since there are no parameters to select for\nprompting, we report the average.\n3.3 Resources\nDatasets. As our multi-label dataset, we use Stack Exchange obtained by the tool of Eleuther-AI to process\nthe data dump on the Internet Archive (EleutherAI, 2020; Stack Exchange, Inc., 2024). As labels, we use\nthe tags assigned to a post by the user – for instance, a user may assign \"baseball\" and \"mlb\" (Major League\nBaseball) to a post about baseball. As our text inputs, we clean the post and append the comments made\nby other users. Since the data format is a set of large XML files, we create an index of all newline characters\nin the bytestream to achieve fast random access. Since there are multiple Stack Exchange fora available, we\nobtain a suite of datasets related to different topics. We chose the following: academia, biology, chemistry,\ncooking, cs, history, law, literature, physics, politics, and sports. For each forum, we plot the cumulative\ndensity functions of label supports after pre-processing in figure 5.\nModels. We run the experiments on Gemma 2 (Team et al., 2024) + Gemma Scope (Lieberum et al., 2024).\nAs a baseline for feature labels, we use the ones provided in Neuronpedia.\nHardware. The entire experiment can be run on a single H100 node within 24 hours for most Stack Exchange\nfora. To properly control the GPU memory consumption, we have used a set of code optimizations, including\npre-allocation and in-place computation of most matrices, and an OOM recovery mechanism that successively\nhalves the batch size if needed. The codebase was originally developed on an RTX4090 machine, and the\nfull pipeline can be run on this GPU for some Stack Exchange fora.\n4 Experimental design and results\nSetup. Using the selected Stack Exchange fora, we apply the pipeline to two different layers in the model\nfor completeness. Here, we use layer 17 and layer 32.3 Our method was originally developed and tuned using\nlayer 32, which we chose in the","truncated":true,"next_offset":20000,"facts":{"authors":["Jørgensen, Mikkel Godsk","Hansen, Lars Kai"],"publisher":"arXiv.org","published_date":"2026-05-29","doi":null,"archived_url":"https://web.archive.org/web/20261027132441/https://arxiv.org/pdf/2605.31183","events":[],"versions":[],"copy_of":[]},"note":"More text follows; call again with offset 20000 to continue."}