On AxBench, SAE steering underperforms both prompting and difference-in-means representations.
First assessment
Read the AxBench TeX source (setup, methods, steering table) and assessed the claim verified at 0.85 confidence with credence 0.92. Kept it atomic, with no decomposition, because it is a single benchmark measurement. Recorded the origin instance plus two mentions (the later papers on output-score and supervised-pipeline selection) and linked the corresponding sibling answers as related. Importance set to 0.4 and contestation to 0.3. Canonical form kept; the assessment states the scope as the benchmark's standard SAE configuration.
Assessed Verified
verdict confidence 0.85 · credence 0.92
The claim restates the headline steering result of AxBench (Wu, Arora et al., Stanford, ICML 2025), and the benchmark paper's own tables bear it out. AxBench steers instruction-tuned Gemma-2-2B (residual stream, layers 10 and 20) and Gemma-2-9B (layers 20 and 31) toward 500 concepts drawn from the Neuronpedia descriptions of Gemma Scope SAE latents. A language-model judge rates each steered response 0 to 2 for concept presence, relevance to the instruction and fluency, and the score is the harmonic mean of the three. The steering strength is chosen per concept on half the instructions and evaluated on the other half. Averaged over the four sites, prompting scored 0.894, difference-in-means (DiffMean) 0.239, the SAE latent whose description named the concept 0.165, and the SAE latent chosen by AUROC on the 144 labelled training examples 0.157. The SAE fell below DiffMean at every one of the four sites, most clearly in earlier layers, and the paper reports the DiffMean advantage as significant, with a 58.7% per-concept win rate against SAEs. Prompting beat the SAE by a wide margin everywhere. The SAE did outperform several other supervised directions (probe, PCA, LAT, supervised steering vector), and every representation method except ReFT-r1 trailed prompting badly. Later work does not dispute this measurement. It changes the method being measured. Arad, Mueller and Belinkov (EMNLP 2025) report that keeping only SAE features with high output scores improves SAE steering two to threefold and makes it competitive with supervised methods. A 2026 preprint by Jørgensen and Hansen reports that features selected and labelled by a supervised pipeline approach LoRA on AxBench. Both results bear on whether SAEs must steer worse, not on whether they did under AxBench's protocol. The claim's general wording ("SAE steering") is best read as the benchmark's standard configuration: description-matched Gemma Scope latents added to the residual stream. For that configuration it is established. It should not be read as a finding that any SAE-based steering method loses on AxBench.
Claim entered the graph