With output-score feature selection, SAE steering is competitive with supervised steering methods on AxBench.
Since this assessment: 1 source added.
Assessment
The evidence weighs against the claim, but the examination is incomplete or the evidence is indirect.
The claim rests on one study, Arad, Mueller and Belinkov's "SAEs Are Good for Steering – If You Select the Right Features" (EMNLP 2025). On AxBench's Concept500 set with instruction-tuned Gemma-2-9B, keeping only sparse autoencoder (SAE) features with an output score of at least 0.1 gave harmonic-mean steering scores of 0.546 at layer 20 and 0.470 at layer 31. The authors set these beside the numbers AxBench originally reported and conclude that filtered SAEs are comparable with supervised methods.
That comparison does not match like with like. The supervised baselines were not re-run. Their scores come from the AxBench paper, which used a gpt-4o-mini judge and its own unreleased sample of instructions. The SAE scores used a Claude 3.7 Sonnet judge and a fresh sample. The authors' own unfiltered SAE run scores 1.5 to 2.8 times AxBench's SAE figure, so pipeline differences alone account for a large part of the apparent gain. The filtered numbers also cover only the concepts whose features pass the threshold, while the baselines cover all concepts. The study does not report what share of concepts were kept.
Even taken at face value, the numbers support only a narrow reading. Filtered SAEs beat the weakly supervised ReFT-r1 at layer 31 (0.470 vs 0.401) and difference-in-means at both layers. They trail ReFT-r1 at layer 20 (0.546 vs 0.630), and they trail the fully supervised LoReFT (0.777, 0.764) and LoRA (0.602, 0.580) at both layers. Prompting, at about 1.07, scores roughly twice as high. The paper's "90.7% of the best method" figure at layer 20 equals 0.546/0.602, which is the ratio to LoRA, not to the best method. An outside evaluation on Gemma-2-2B layer 20, run under the AxBench protocol (arXiv 2609.35367), found that the output-score filter does not give the reported gains. That test did not cover the 9B setting.
What the evidence does support is that output-score selection makes SAE features much better steering vectors than unselected ones, closing part of AxBench's gap. A matched re-run would settle the stronger claim: all methods scored on the same concepts, instructions and judge, with the retained fraction reported.
Full reasoning: the evidence and decisions behind this verdict
Read the source's TeX (arXiv 2505.20063, final_version.tex), Section 4.4, Table 1 and Appendix on AxBench. - Same model, layers and concept set as AxBench's 9B setting (Gemma-2-9B-it, L20/L31, Concept500), same three metrics and harmonic mean. Differences: judge is Claude 3.7 Sonnet (AxBench used gpt-4o-mini, confirmed by arXiv 2609.35367); 10 instructions re-sampled because AxBench did not release its sample (footnote); steering-factor selection on 5 held-out instructions per concept with the authors' own criterion. - Baselines in Table 1 are labelled "AxBench reported results (Wu et al. 2025)": not re-run. - Unfiltered SAE in the authors' pipeline: 0.293 (L20), 0.387 (L31), against AxBench's SAE 0.191 and 0.140. Ratios 1.53 and 2.76. The paper's "nearly threefold improvement" over AxBench (0.546/0.191 = 2.86) therefore mixes the filter's effect with pipeline differences. The like-for-like filter effect is 0.293 to 0.546 (L20) and 0.387 to 0.470 (L31), and it is not monotonic in threshold (L20: 0.338 at 0.01 < 0.373 at 0.001). - Filtering removes features, and each feature belongs to one concept, so the filtered score averages over a subset of concepts. The number kept is not reported. - Ratios at face value: L20 0.546/0.630 (ReFT-r1) = 0.867, /0.602 (LoRA) = 0.907, /0.777 (LoReFT) = 0.703, /1.075 (Prompt) = 0.508. L31 0.470/0.401 (ReFT-r1) = 1.17, /0.580 (LoRA) = 0.81, /0.764 (LoReFT) = 0.615, /1.072 = 0.438. The paper's 90.7% therefore matches the LoRA ratio, not the "best method" in its sentence. - Outside check: arXiv 2609.35367 (DAFI), Appendix N. It evaluated the output-score filter on Gemma-2-2B L20 under the AxBench protocol with a gpt-4o-mini judge, found the filter retains pairs the judge scores low, and did not observe the gains. It explicitly does not speak to the 9B protocol. No independent matched re-run on 9B was found. A citing paper (arXiv 2602.10437) summarises the result as narrowing the gap, not closing it. - "Competitive with supervised steering methods" read plainly includes LoReFT and LoRA, which AxBench treats as the supervised fine-tuning baselines. Filtered SAEs are not competitive with these even on the uncorrected numbers. Against ReFT-r1 the results are mixed and confounded. Credence 0.2 allows for a matched re-run showing near-parity with the weaker representation-based baselines.
Decomposition
How this claim breaks down: each argument is stated as it runs, with its subclaims linked inline. ↗︎ opens a subclaim; the map shows how they fit together.
On AxBench Concept500 with instruction-tuned Gemma-2-9B, keeping only SAE features with output score at least 0.1 raises their harmonic-mean steering score to 0.546 (layer 20) and 0.470 (layer 31) (output-score selection improves SAE steering). AxBench reported 0.630 and 0.401 for the weakly supervised ReFT-r1 and 0.322 and 0.158 for difference-in-means on the same layers. Taking these numbers as comparable, filtered SAEs match or beat the best representation-based supervised steering methods, so they are competitive with supervised steering.
- Does it follow?
- slight even if the premises holdEven if the premises are true, they make the conclusion only a little more likely.constitution →
- Do the premises hold?
- 1 not separately assessed
Granting the numbers, the inference is weak. It treats scores from different judges, instruction samples and concept subsets as comparable. The authors' own unfiltered run shows those differences alone raise SAE scores 1.5 to 2.8 times. Even at face value, filtered SAEs beat only ReFT-r1 at one layer and difference-in-means, and they trail LoReFT, LoRA and prompting. The argument does establish part of its premise: output-score selection improves SAE steering within one pipeline. That is a real but narrower result.
Provenance
Where this claim has been said, linked to its canonical form.
Neither source has been traced to an origin, or to one another.
These re- sults demonstrates that with effective feature selec- tion SAEs are comparable with existing methods, including supervised or weakly-supervised meth- ods which require concept-specific datasets.
Section 4.4, AxBench evaluation on instruction-tuned Gemma-2-9B, Concept500, layers 20 and 31
The assertion outruns the source's own evidence. The paper's own Table 1 sets the filtered SAE scores beside baseline numbers copied from AxBench, which used a different judge and a different instruction sample. The authors' own unfiltered SAE run scores well above AxBench's SAE figure, so part of the apparent gain comes from the pipeline. Even so, the filtered SAEs trail LoReFT, LoRA and prompting at both layers. The 90.7 percent figure matches the ratio to LoRA, not to the best method.
Mentioned in
Sources that refer to the claim without taking a side: they cite it, report the debate, or raise it in passing.
We evaluate the output-score filter of Arad et al. (2025) on Gemma-2-2B under the AxBench protocol and do not observe the gains they report on Gemma-2-9B under their own protocol.
Appendix N. An outside check of the output-score filter, on a different model, under AxBench's gpt-4o-mini judge.
Cite this claim: a formal citation with its evidence attached
Embed this claim: show it on another site with its current assessment
An embed reports the graph's judgment as it stands when the page is viewed: the status with its confidence, how disputed the claim is, and the date. It links back here.
<blockquote class="minerval-claim" data-claim-id="d3311aac-502f-4e69-ab97-53ee2b6de622"><a href="https://minerval.ai/claims/d3311aac-502f-4e69-ab97-53ee2b6de622">With output-score feature selection, SAE steering is competitive with supervised steering methods on AxBench.</a></blockquote> <script async src="https://minerval.ai/embed.js"></script>
<iframe src="https://minerval.ai/embed/claims/d3311aac-502f-4e69-ab97-53ee2b6de622" title="Minerval: With output-score feature selection, SAE steering is competitive with supervised steering…" width="480" height="200" style="border:0;max-width:100%" loading="lazy"></iframe>
[](https://minerval.ai/claims/d3311aac-502f-4e69-ab97-53ee2b6de622)
The badge now: 
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
Stated views None yet. A view stated to the graph is shown on this page in its speaker’s words, as a source like any other, and carries no weight in the assessment.
The attention this claim received was paid for by a funded mandate. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by extractor · Nov 13, 2026. Every judgment on this page is accompanied by a reasoning trace.