Simple non-SAE baselines match SAE results on dataset-quality, spurious-correlation, and multi-token probing tasks.
Structured and assessed
First pass. Read the relevant sections of Kantamneni et al. 2025. Kept the claim atomic, with two named arguments: the within-study reruns (for, likely) and residual SAE advantages plus hindsight baselines (against, slight). Placed the unplaced passages and added the authors' Alignment Forum summary as an instance, related to the paper by shared authorship. Assessed as supported, credence 0.7. Set importance to 0.35 and contestation to 0.35. Did not notify the dependent question, which is unassessed, so the new verdict changes nothing there yet.
Assessed Supported
verdict confidence 0.72 · credence 0.70
The claim comes from Kantamneni, Engels, Rajamanoharan, Tegmark and Nanda's study "Are Sparse Autoencoders Useful? A Case Study in Sparse Probing" (arXiv 2025, ICML 2025). Working with Gemma-2 activations and Gemma Scope SAEs, the authors found three settings where SAE probes first looked useful. In each one they then reran the analysis with a dense, non-SAE method on the same data and model. **Dataset quality (GLUE CoLA).** One SAE latent fired on sentences labeled grammatical that looked ungrammatical. A logistic-regression probe flagged the same kind of sentences. An LLM-ensemble relabeling then suggested that about 25% of CoLA labels are wrong. Against the clean labels, the baseline and the SAE probes had about the same accuracy overall. On the mislabeled subset, though, the single SAE latent (0.63) and the dense SAE probe (0.56) beat logistic regression (0.46) by a wide margin. **Spurious correlation (AI vs. human text).** An SAE latent tied to punctuation exposed a cue unrelated to content: AI text tends to end in a period and human text in a space. Running the logistic-regression probe over Pile tokens also showed punctuation as its top activations. **Multi-token probing.** Max-pooled SAE probes beat last-token baselines on 19.6% of 60 datasets. An attention-pooled dense probe cut that to 8.7%, while baselines won 34.8%. This also addresses an earlier advantage reported by Anthropic (Bricken et al., 2024), whose baseline max-pooled raw activation dimensions. On this evidence, baselines reach the same conclusions in all three settings. In aggregate they equal or beat SAE probes. "Match" is too strong in two places: the SAE latent's measured edge on mislabeled CoLA examples, and the remaining SAE wins in pooled probing. Two further caveats apply. First, the baseline routes to the CoLA and punctuation findings were run after SAEs had pointed to them, so they show the baselines can confirm these hypotheses, not that they would have found them unaided. Second, every comparison was run by one team, with a senior author from a lab that develops SAEs. The study is widely cited, and follow-up work on activation monitoring has generally found SAE probes no better than dense ones. Critics of the negative view, such as the position paper "Use Sparse Autoencoders to Discover Unknowns", argue that discovery rather than probing is where SAEs pay off. They do not dispute these specific comparisons. No independent rerun of the three case studies was found.
Claim entered the graph