Step 21: assistant

text

I've rated the claim **supported**: about 70% likely true, with 0.72 confidence in that rating. This was its first pass, and it used about $0.61 of the $1.20 budget.

**Where the comparisons come from.** For all three task families, the SAE result and the non-SAE baseline come from the same study: Kantamneni, Engels, Rajamanoharan, Tegmark and Nanda, "Are Sparse Autoencoders Useful? A Case Study in Sparse Probing" (2025, published at ICML 2025). So the data and model are the same by construction: Gemma-2 activations with Gemma Scope SAEs. The mandate asked for separate studies, but for these three tasks there aren't any. The one outside study involved is an earlier Anthropic result (Bricken et al. 2024, below).

| Task family | Baseline used | How close the match is |
|---|---|---|
| Dataset quality (CoLA, a grammaticality dataset) | Plain logistic regression | Against cleaned labels (about 25% of the originals judged wrong), overall accuracy was about equal. On the mislabeled examples, the SAE methods were clearly better: a single SAE feature scored 0.63 and an SAE probe 0.56, against 0.46 for logistic regression. |
| Spurious correlation (AI vs. human text) | Running the logistic-regression probe over a large text corpus and checking which tokens it fires on most | Same conclusion, but only qualitative: AI text tends to end in a period, human text in a space. |
| Multi-token probing | A dense probe that pools across tokens with attention | SAE probes beat simple last-token baselines on 19.6% of 60 datasets. Against the pooled dense probe that fell to 8.7%, while the baselines won on 34.8%. |

**Why "supported" and not "verified":**
- **Not always a match.** The baselines reach the same conclusions in all three cases and equal or beat SAE probes overall. But in two places "match" overstates it: the SAE edge on the mislabeled CoLA examples and the 8.7% of datasets where pooled SAE probes still win.
- **Baselines run in hindsight.** The baseline checks for CoLA and the punctuation cue were run after SAEs had already pointed to the answer. They show the baselines can confirm these findings, not that they would have found them unaided.
- **One team.** Every comparison was run by the same authors, whose senior author is at a lab that builds SAEs, and I found no independent rerun of these three case studies.

**Replies.** The paper addresses Anthropic's earlier positive SAE probing result (Bricken et al. 2024) by arguing that its baseline was too weak; that result's dataset wasn't available to check. I found no rebuttal from SAE proponents. A position paper, "Use Sparse Autoencoders to Discover Unknowns", accepts these probing results and argues instead that SAEs earn their keep at finding unknown concepts, not at probing.

**What I put on the claim page:**
- Two arguments: the reruns within the study (for; makes the claim much more likely if reported faithfully) and the remaining SAE advantages plus the hindsight baselines (against; bears only slightly against the claim as worded).
- The authors' own Alignment Forum summary as a second source.
- A note that the paper's abstract slightly overstates what its own numbers show.
- Importance and contestation both set to 0.35.

I did not notify the one node that depends on this claim, a question about when SAE features beat simpler methods, because it hasn't been assessed yet. I estimate that another pass would add little (0.25 on the 0–1 scale). The thing that would change the verdict is an independent rerun of these case studies.