Step 15: assistant

Calls provenance_record_source_relationship

{
  "reasoning": "The Alignment Forum post is the paper authors' own summary of the same work.",
  "confidence": 0.9,
  "relation_type": "shares_authorship",
  "child_source_id": "1bc0f640-83f8-4d49-8709-5ce9b35780a2",
  "parent_source_id": "b45014e2-63ec-41af-8a6f-59db01b1e0fc"
}

Calls update_claim_assessment

{
  "status": "supported",
  "claim_id": "738d9786-0b87-4b75-9c33-cabe43c903d5",
  "assessment": "The claim comes from Kantamneni, Engels, Rajamanoharan, Tegmark and Nanda's study \"Are Sparse Autoencoders Useful? A Case Study in Sparse Probing\" (arXiv 2025, ICML 2025). Working with Gemma-2 activations and Gemma Scope SAEs, the authors found three settings where SAE probes first looked useful. In each one they then reran the analysis with a dense, non-SAE method on the same data and model.\n\n**Dataset quality (GLUE CoLA).** One SAE latent fired on sentences labeled grammatical that looked ungrammatical. A logistic-regression probe flagged the same kind of sentences. An LLM-ensemble relabeling then suggested that about 25% of CoLA labels are wrong. Against the clean labels, the baseline and the SAE probes had about the same accuracy overall. On the mislabeled subset, though, the single SAE latent (0.63) and the dense SAE probe (0.56) beat logistic regression (0.46) by a wide margin.\n\n**Spurious correlation (AI vs. human text).** An SAE latent tied to punctuation exposed a cue unrelated to content: AI text tends to end in a period and human text in a space. Running the logistic-regression probe over Pile tokens also showed punctuation as its top activations.\n\n**Multi-token probing.** Max-pooled SAE probes beat last-token baselines on 19.6% of 60 datasets. An attention-pooled dense probe cut that to 8.7%, while baselines won 34.8%. This also addresses an earlier advantage reported by Anthropic (Bricken et al., 2024), whose baseline max-pooled raw activation dimensions.\n\nOn this evidence, baselines reach the same conclusions in all three settings. In aggregate they equal or beat SAE probes. \"Match\" is too strong in two places: the SAE latent's measured edge on mislabeled CoLA examples, and the remaining SAE wins in pooled probing. Two further caveats apply. First, the baseline routes to the CoLA and punctuation findings were run after SAEs had pointed to them, so they show the baselines can confirm these hypotheses, not that they would have found them unaided. Second, every comparison was run by one team, with a senior author from a lab that develops SAEs. The study is widely cited, and follow-up work on activation monitoring has generally found SAE probes no better than dense ones. Critics of the negative view, such as the position paper \"Use Sparse Autoencoders to Discover Unknowns\", argue that discovery rather than probing is where SAEs pay off. They do not dispute these specific comparisons. No independent rerun of the three case studies was found.",
  "confidence": 0.72,
  "claim_credence": 0.7,
  "marginal_yield": 0.25,
  "reasoning_trace": "The authors read Sections 4.3 and 5 and Figures 9 and 11 of arXiv 2502.16681 in full. The SAE result and the baseline comparison are in the same paper, so data, model (Gemma-2 with Gemma Scope SAEs) and splits are shared by construction. How the baselines were built: plain L2 logistic regression on last-token activations for CoLA and the punctuation case. For multi-token, an attention-pooled probe softmax_t(X_t·q)·X·v, with a quiver-of-arrows choice between pooled and last-token for both methods (Section 5, Eq. 2). How close the match is: CoLA accuracy on clean labels is about equal across methods (0.72 to 0.81 range in Figure 9). On the mislabeled subset it is 0.63 (latent), 0.56 (SAE k=128) and 0.46 (baseline), an SAE advantage. The punctuation case is qualitative, the same conclusion via Pile max-activating tokens. Multi-token: the SAE win rate (>0.005 AUC) falls from 19.6% to 8.7%, against 34.8% baseline wins. The earlier external SAE probing result (Bricken et al. 2024, Anthropic) is addressed only indirectly, since its dataset was unavailable. Gallifant et al. 2025 report SAE multi-token probes beating single-token baselines only. Responses: a web search found no rebuttal of these case studies by SAE proponents. Peng et al.'s 2025 position paper accepts the probing results and reframes SAE value as discovery. Tillman & Mossing (arXiv 2504.20271) independently found SAE-based monitoring no better and less robust, on a different task. Credence 0.7 reflects strong support for \"comparable conclusions\", discounted for the measured residual SAE advantages and the single-team, post-hoc character of the baselines."
}