Simple non-SAE baselines match SAE results on dataset-quality, spurious-correlation, and multi-token probing tasks.
Assessment
The evidence favors the claim, but the examination is incomplete or the evidence is indirect.
The claim comes from Kantamneni, Engels, Rajamanoharan, Tegmark and Nanda's study "Are Sparse Autoencoders Useful? A Case Study in Sparse Probing" (arXiv 2025, ICML 2025). Working with Gemma-2 activations and Gemma Scope SAEs, the authors found three settings where SAE probes first looked useful. In each one they then reran the analysis with a dense, non-SAE method on the same data and model.
**Dataset quality (GLUE CoLA).** One SAE latent fired on sentences labeled grammatical that looked ungrammatical. A logistic-regression probe flagged the same kind of sentences. An LLM-ensemble relabeling then suggested that about 25% of CoLA labels are wrong. Against the clean labels, the baseline and the SAE probes had about the same accuracy overall. On the mislabeled subset, though, the single SAE latent (0.63) and the dense SAE probe (0.56) beat logistic regression (0.46) by a wide margin.
**Spurious correlation (AI vs. human text).** An SAE latent tied to punctuation exposed a cue unrelated to content: AI text tends to end in a period and human text in a space. Running the logistic-regression probe over Pile tokens also showed punctuation as its top activations.
**Multi-token probing.** Max-pooled SAE probes beat last-token baselines on 19.6% of 60 datasets. An attention-pooled dense probe cut that to 8.7%, while baselines won 34.8%. This also addresses an earlier advantage reported by Anthropic (Bricken et al., 2024), whose baseline max-pooled raw activation dimensions.
On this evidence, baselines reach the same conclusions in all three settings. In aggregate they equal or beat SAE probes. "Match" is too strong in two places: the SAE latent's measured edge on mislabeled CoLA examples, and the remaining SAE wins in pooled probing. Two further caveats apply. First, the baseline routes to the CoLA and punctuation findings were run after SAEs had pointed to them, so they show the baselines can confirm these hypotheses, not that they would have found them unaided. Second, every comparison was run by one team, with a senior author from a lab that develops SAEs. The study is widely cited, and follow-up work on activation monitoring has generally found SAE probes no better than dense ones. Critics of the negative view, such as the position paper "Use Sparse Autoencoders to Discover Unknowns", argue that discovery rather than probing is where SAEs pay off. They do not dispute these specific comparisons. No independent rerun of the three case studies was found.
Full reasoning: the evidence and decisions behind this verdict
The authors read Sections 4.3 and 5 and Figures 9 and 11 of arXiv 2502.16681 in full. The SAE result and the baseline comparison are in the same paper, so data, model (Gemma-2 with Gemma Scope SAEs) and splits are shared by construction. How the baselines were built: plain L2 logistic regression on last-token activations for CoLA and the punctuation case. For multi-token, an attention-pooled probe softmax_t(X_t·q)·X·v, with a quiver-of-arrows choice between pooled and last-token for both methods (Section 5, Eq. 2). How close the match is: CoLA accuracy on clean labels is about equal across methods (0.72 to 0.81 range in Figure 9). On the mislabeled subset it is 0.63 (latent), 0.56 (SAE k=128) and 0.46 (baseline), an SAE advantage. The punctuation case is qualitative, the same conclusion via Pile max-activating tokens. Multi-token: the SAE win rate (>0.005 AUC) falls from 19.6% to 8.7%, against 34.8% baseline wins. The earlier external SAE probing result (Bricken et al. 2024, Anthropic) is addressed only indirectly, since its dataset was unavailable. Gallifant et al. 2025 report SAE multi-token probes beating single-token baselines only. Responses: a web search found no rebuttal of these case studies by SAE proponents. Peng et al.'s 2025 position paper accepts the probing results and reframes SAE value as discovery. Tillman & Mossing (arXiv 2504.20271) independently found SAE-based monitoring no better and less robust, on a different task. Credence 0.7 reflects strong support for "comparable conclusions", discounted for the measured residual SAE advantages and the single-team, post-hoc character of the baselines.
Decomposition
This claim is atomic: it bottoms out in a bedrock fact, an open empirical question, or a value premise, and does not decompose further.
Provenance
Where this claim has been said, linked to its canonical form.
Neither source has been traced to an origin, or to one another.
Additionally, although SAEs initially appear promising for identifying spurious correla- tions, detecting poor dataset quality, and training multi-token probes, we are able to achieve simi- lar results with simple non-SAE baselines as well.
Abstract; case studies on GLUE CoLA label errors and a spurious punctuation feature in AI-vs-human text.
The assertion outruns the source's own evidence. The body supports the abstract for the conclusions reached: dense baselines flag the same CoLA errors, point to the same punctuation cue, and largely close the pooled-probe gap. It slightly overstates on numbers: on the mislabeled CoLA subset the SAE latent (0.63) and dense SAE probe (0.56) beat logistic regression (0.46), pooled SAE probes still win 8.7% of datasets, and the baseline checks were run after SAEs had suggested the hypotheses. All comparisons are by one team on Gemma-2 SAEs. Worth reading closely: Sections 4.3 and 5, with Figures 9 and 11, hold all the numbers the claim rests on.
However, when we implemented an attention-pooled baseline that could also leverage information across tokens, the SAE probe win rate dropped to 8.7%.
The authors' own summary of the paper; same voice as the preprint, covering the multi-token case.
How these sources relate
- Takeaways From Our Recent Work on SAE Probing and Are Sparse Autoencoders Useful? A Case Study in Sparse Probing share an author. The Alignment Forum post is the paper authors' own summary of the same work.
Cite this claim: a formal citation with its evidence attached
Embed this claim: show it on another site with its current assessment
An embed reports the graph's judgment as it stands when the page is viewed: the status with its confidence, how disputed the claim is, and the date. It links back here.
<blockquote class="minerval-claim" data-claim-id="738d9786-0b87-4b75-9c33-cabe43c903d5"><a href="https://minerval.ai/claims/738d9786-0b87-4b75-9c33-cabe43c903d5">Simple non-SAE baselines match SAE results on dataset-quality, spurious-correlation, and multi-token probing tasks.</a></blockquote> <script async src="https://minerval.ai/embed.js"></script>
<iframe src="https://minerval.ai/embed/claims/738d9786-0b87-4b75-9c33-cabe43c903d5" title="Minerval: Simple non-SAE baselines match SAE results on dataset-quality, spurious-correlation, and…" width="480" height="225" style="border:0;max-width:100%" loading="lazy"></iframe>
[](https://minerval.ai/claims/738d9786-0b87-4b75-9c33-cabe43c903d5)
The badge now: 
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
Stated views None yet. A view stated to the graph is shown on this page in its speaker’s words, as a source like any other, and carries no weight in the assessment.
The attention this claim received was paid for by a funded mandate. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by extractor · Nov 13, 2026. Every judgment on this page is accompanied by a reasoning trace.