Sparse autoencoders rarely outperform simple baselines on practical downstream interpretability tasks.
Structured and assessed
First pass on small budget. Linked sibling answers as basis: probing null (supports), AxBench steering loss (supports), SAE-space jailbreak steering win (contradicts). No named arguments needed. Dismissed one unplaced passage (covered by basis). Importance 0.55, contestation 0.55. Assessed supported, credence 0.72, tallied by task family per the plan; unlearning and exploratory-task evidence from background knowledge, flagged for a later pass.
Assessed Supported
verdict confidence 0.70 · credence 0.72
The evidence favours the claim for the task families where designed comparisons against tuned baselines exist, which are mainly concept detection (probing) and steering, with a smaller body on unlearning. It is thinner and more favourable to sparse autoencoders (SAEs) on exploratory tasks such as auditing, model diffing and hypothesis generation, where clear baselines are harder to define. **Probing.** The largest comparison, a 113-dataset study on Gemma-2-9B, found that adding SAE probes to a toolkit of tuned baselines did not improve results under scarce data, class imbalance, label noise or covariate shift; earlier apparent wins disappeared against the stronger toolkit (SAE probes show no consistent advantage over baseline probes). Pooled SAE probes beat single-token raw probes on multi-token text, but most of that advantage vanishes once the raw-activation probe is also pooled, leaving a small residual of a few datasets, mostly within noise. Google DeepMind found SAE probes worse out of distribution on harmful-intent detection. Reported detection wins (an OpenAI moderation study, a Goodfire and Rakuten personal-data report) mostly compare against unmatched or weaker baselines, or come from groups that build SAE tooling and are unreplicated. **Steering.** On AxBench, description-matched SAE features steer worse than difference-of-means and far worse than prompting. Follow-up work recovers rough parity by choosing features with labels or by measured output effect, but does not show a general win. The one direct win located is SAE-space mean-shift steering beating dense mean-shift for jailbreak mitigation, especially on held-out attacks: a single unreplicated 2026 study. **Unlearning.** Work cited in the Google DeepMind update (Farrell et al.) found SAE-based unlearning less effective than established fine-tuning methods; some later SAE-based guardrail methods claim gains, but these have not been independently confirmed. **Auditing, model diffing, hypothesis generation.** Here SAEs have their most credible practical uses: auditors with SAE access found a planted hidden objective in Anthropic's auditing game, crosscoder and SAE diffing has surfaced chat-tuning-specific features, and SAE-based hypothesis generation has been reported to beat prompting-based baselines on text datasets. These tasks seldom have a tuned simple baseline run alongside, and in the auditing game simpler search over training data also worked, so they show usefulness more than a measured advantage. Taken together, losses or ties against well-tuned baselines dominate where the comparison has been run, and the documented wins are few, narrow, mostly from interested groups and largely unreplicated. That fits "rarely". The claim would weaken if matched, independently replicated wins accumulated, for example label-selected SAE features under distribution shift, or exploratory tasks such as auditing with a proper baseline arm.
Claim entered the graph