On AxBench, SAE steering underperforms both prompting and difference-in-means representations.
Assessment
The evidence, examined directly, establishes the claim.
The claim restates the headline steering result of AxBench (Wu, Arora et al., Stanford, ICML 2025), and the benchmark paper's own tables bear it out. AxBench steers instruction-tuned Gemma-2-2B (residual stream, layers 10 and 20) and Gemma-2-9B (layers 20 and 31) toward 500 concepts drawn from the Neuronpedia descriptions of Gemma Scope SAE latents. A language-model judge rates each steered response 0 to 2 for concept presence, relevance to the instruction and fluency, and the score is the harmonic mean of the three. The steering strength is chosen per concept on half the instructions and evaluated on the other half.
Averaged over the four sites, prompting scored 0.894, difference-in-means (DiffMean) 0.239, the SAE latent whose description named the concept 0.165, and the SAE latent chosen by AUROC on the 144 labelled training examples 0.157. The SAE fell below DiffMean at every one of the four sites, most clearly in earlier layers, and the paper reports the DiffMean advantage as significant, with a 58.7% per-concept win rate against SAEs. Prompting beat the SAE by a wide margin everywhere. The SAE did outperform several other supervised directions (probe, PCA, LAT, supervised steering vector), and every representation method except ReFT-r1 trailed prompting badly.
Later work does not dispute this measurement. It changes the method being measured. Arad, Mueller and Belinkov (EMNLP 2025) report that keeping only SAE features with high output scores improves SAE steering two to threefold and makes it competitive with supervised methods. A 2026 preprint by Jørgensen and Hansen reports that features selected and labelled by a supervised pipeline approach LoRA on AxBench. Both results bear on whether SAEs must steer worse, not on whether they did under AxBench's protocol. The claim's general wording ("SAE steering") is best read as the benchmark's standard configuration: description-matched Gemma Scope latents added to the residual stream. For that configuration it is established. It should not be read as a finding that any SAE-based steering method loses on AxBench.
Full reasoning: the evidence and decisions behind this verdict
Read directly from the TeX source of arXiv 2501.17148 (main.tex: abstract, introduction, setup, methods, results).
Setup. 500 concepts (Concept500) sampled from the Neuronpedia/Gemma Scope concept list. Sites: Gemma-2-2B-it L10 and L20 (using Gemma Scope SAEs trained on the base model, since none existed for the instruction-tuned model), and Gemma-2-9B-it L20 and L31. Training data: 144 synthetic examples per concept generated by gpt-4o-mini, balanced positives and negatives. SAE method: the Gemma Scope JumpReLU latent corresponding to the concept label, with its decoder vector added to the residual stream (activation addition, as for DiffMean). The magnitude is the steering factor times the latent's maximum activation from Neuronpedia; clamping is ablated in an appendix. SAE-A picks the latent with the highest AUROC on the training labels, using max-pooled activations. DiffMean is the unit-normalised mean of positive minus negative token activations. Prompting uses an LLM to engineer a steering prompt prepended to the instruction. Steering metric: an LLM judge scores concept, instruct and fluency each 0, 1 or 2, and the score is their harmonic mean. Ten Alpaca-Eval instructions per concept, 14 steering factors, up to 128 tokens; the factor is selected on one half of the instructions and the result reported on the other half.
Results (Table "Mean overall steering scores"): Prompt 0.698 / 0.731 / 1.075 / 1.072 (average 0.894); DiffMean 0.297 / 0.178 / 0.322 / 0.158 (0.239); SAE 0.177 / 0.151 / 0.191 / 0.140 (0.165); SAE-A 0.166 / 0.132 / 0.186 / 0.143 (0.157). The text says "DiffMean significantly outperforms SAEs, particularly in earlier layers". Win rate of DiffMean against SAE: 63.2 / 55.2 / 64.3 / 52.2%, average 58.7%. Prompting's win rate against SAE: 94.5%. The SAE beat Probe (0.098), PCA (0.105), LAT (0.127) and SSV (0.026). Concept detection, for context: SAE AUROC 0.695 and SAE-A 0.917, against DiffMean 0.942.
Margins. Against prompting the gap is very large. Against DiffMean the absolute gap is modest (0.074 on a 0 to 2 scale), and at 9B L31 it is small (0.158 against 0.140, win rate 52.2%). So "underperforms difference-in-means" holds on average and at each site, but is weakest at late layers. Credence is held a little below 1 for this reason and for the dependence on one LLM judge and one protocol.
Later papers, read at abstract level only. Arad et al. 2505.20063 change the selection (output-score filtering) and report two- to threefold gains and competitiveness with supervised methods. Jørgensen and Hansen 2605.31183 call themselves a "partial rebuttal" and use a supervised selection and labelling pipeline. The question node's assessment, from earlier research, records that this work steers only concepts with a well-matched feature, compares against the original paper's baseline figures on a different concept set, and concedes it is not competitive with prompting. Neither paper reports re-running the original configuration and getting different numbers. Chanin (2026) cites the result as found. No source located disputes the measurement itself. The full results tables of the two rebuttals were not read this pass, so the size of their improvements is taken from their abstracts.
Reading of the claim. The claim names no feature-selection method. Its natural reading is the AxBench result as reported. Taken as "no SAE steering method can beat DiffMean on AxBench", it would be weaker, given the rebuttals. The assessment states the scope explicitly.
Decomposition
This claim is atomic: it bottoms out in a bedrock fact, an open empirical question, or a value premise, and does not decompose further.
Provenance
Where this claim has been said, linked to its canonical form.
None of its 4 sources has been traced to an origin, or to one another.
On steering, only ReFT-r1 is competitive with finetuning and prompting baselines, while SAEs fall behind both ReFT-r1 and difference-in-means on both axes.
Introduction of the benchmark paper; the origin of the result. Table of mean overall steering scores: Prompt 0.894, DiffMean 0.239, SAE 0.165, SAE-A 0.157.
Wu et al. [32] find SAE steering inferior to prompting and difference-in-means representations.
Related work, 'Cases where supervised baselines beat SAEs' (AxBench)
Mentioned in
Sources that refer to the claim without taking a side: they cite it, report the debate, or raise it in passing.
after filtering out features with low output scores, we obtain 2-3x improvements when steering with SAEs, making them competitive with supervised methods.
Responds to the AxBench result by changing how SAE features are selected (output-score filtering), not by disputing the original measurement.
This work serves as a partial rebuttal for Sparse Autoencoders and suggests that the results of Wu et al. (2025) did not do them full justice. We find that Sparse Autoencoders can, in fact, perform close to on par with the reference LoRA performance on the AxBench benchmark, when features are selected and labelled with our supervised pipeline.
Partial rebuttal that changes the SAE feature selection and labelling method; it does not report a re-run of the original configuration that contradicts the published numbers.
Cite this claim: a formal citation with its evidence attached
Embed this claim: show it on another site with its current assessment
An embed reports the graph's judgment as it stands when the page is viewed: the status with its confidence, how disputed the claim is, and the date. It links back here.
<blockquote class="minerval-claim" data-claim-id="7c1050dc-7ca9-492e-8d3f-ade8d793de4c"><a href="https://minerval.ai/claims/7c1050dc-7ca9-492e-8d3f-ade8d793de4c">On AxBench, SAE steering underperforms both prompting and difference-in-means representations.</a></blockquote> <script async src="https://minerval.ai/embed.js"></script>
<iframe src="https://minerval.ai/embed/claims/7c1050dc-7ca9-492e-8d3f-ade8d793de4c" title="Minerval: On AxBench, SAE steering underperforms both prompting and difference-in-means representat…" width="480" height="200" style="border:0;max-width:100%" loading="lazy"></iframe>
[](https://minerval.ai/claims/7c1050dc-7ca9-492e-8d3f-ade8d793de4c)
The badge now: 
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
Stated views None yet. A view stated to the graph is shown on this page in its speaker’s words, as a source like any other, and carries no weight in the assessment.
The attention this claim received was paid for by a funded mandate. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by extractor · Nov 13, 2026. Every judgment on this page is accompanied by a reasoning trace.