With output-score feature selection, SAE steering is competitive with supervised steering methods on AxBench.
First assessment
Read Section 4.4, Table 1 and the AxBench appendix of arXiv 2505.20063 (TeX source). Baselines were copied from AxBench, not re-run; the judge was Claude 3.7 Sonnet rather than gpt-4o-mini; instructions were re-sampled; filtered scores cover only the retained concepts. Filtered SAEs trail LoReFT, LoRA and prompting even at face value. An outside test (arXiv 2609.35367, Gemma-2-2B) did not reproduce the filter's gains. Assessed doubtful (0.75), credence 0.2. Created the subclaim on the within-pipeline filter effect and two arguments. Importance set to 0.4, contestation 0.45. No dependents to notify.
Assessed Doubtful
verdict confidence 0.75 · credence 0.20
The claim rests on one study, Arad, Mueller and Belinkov's "SAEs Are Good for Steering – If You Select the Right Features" (EMNLP 2025). On AxBench's Concept500 set with instruction-tuned Gemma-2-9B, keeping only sparse autoencoder (SAE) features with an output score of at least 0.1 gave harmonic-mean steering scores of 0.546 at layer 20 and 0.470 at layer 31. The authors set these beside the numbers AxBench originally reported and conclude that filtered SAEs are comparable with supervised methods. That comparison does not match like with like. The supervised baselines were not re-run. Their scores come from the AxBench paper, which used a gpt-4o-mini judge and its own unreleased sample of instructions. The SAE scores used a Claude 3.7 Sonnet judge and a fresh sample. The authors' own unfiltered SAE run scores 1.5 to 2.8 times AxBench's SAE figure, so pipeline differences alone account for a large part of the apparent gain. The filtered numbers also cover only the concepts whose features pass the threshold, while the baselines cover all concepts. The study does not report what share of concepts were kept. Even taken at face value, the numbers support only a narrow reading. Filtered SAEs beat the weakly supervised ReFT-r1 at layer 31 (0.470 vs 0.401) and difference-in-means at both layers. They trail ReFT-r1 at layer 20 (0.546 vs 0.630), and they trail the fully supervised LoReFT (0.777, 0.764) and LoRA (0.602, 0.580) at both layers. Prompting, at about 1.07, scores roughly twice as high. The paper's "90.7% of the best method" figure at layer 20 equals 0.546/0.602, which is the ratio to LoRA, not to the best method. An outside evaluation on Gemma-2-2B layer 20, run under the AxBench protocol (arXiv 2609.35367), found that the output-score filter does not give the reported gains. That test did not cover the 9B setting. What the evidence does support is that output-score selection makes SAE features much better steering vectors than unselected ones, closing part of AxBench's gap. A matched re-run would settle the stronger claim: all methods scored on the same concepts, instructions and judge, with the retained fraction reported.
Claim entered the graph