Step 25: assistant

Calls update_claim_assessment

{
  "status": "doubtful",
  "claim_id": "20fee30f-0192-4b8c-b479-83c3f230014d",
  "assessment": "The claim states a hypothesis that motivated several 2024 to 2025 studies: a probe restricted to a few interpretable sparse-autoencoder (SAE) latents should latch onto the target concept and ignore minor spurious cues that a dense linear probe can exploit. The tests run so far, all on Gemma Scope and Llama Scope SAEs, found the opposite. In Google DeepMind's harmful-intent experiment on Gemma 2 9B, k-sparse SAE probes (about k=20) fit the training and in-distribution test sets but transferred worse to held-out jailbreaks and benign transformations than a dense probe, which scored near-perfect AUROC out of distribution. Kantamneni et al. (2025), across 113 datasets, found that SAE probes [[claim:0cfd9ad2-65a2-49ef-8fa2-5f035e22073f|underperform tuned logistic regression under covariate shift and label noise]], and that pruning latents judged spurious recovered only a small part of the out-of-distribution drop.\n\nThe evidence is indirect in one important respect: no study located here induced a controlled spurious correlation, such as a cue perfectly confounded with the label, and compared worst-group or decorrelated accuracy for sparse SAE probes against dense probes. The DeepMind team removed the spurious formatting cues it found before training, which may favour the dense probe; that cleanup used SAE latents, though Kantamneni et al. show simple non-SAE methods find the same cues. The qualifier \"sufficiently good\" also limits refutation, since every test used current SAEs that are incomplete and lose task-relevant information. Even for an ideal SAE, the argument needs the target concept to be more cleanly isolated in latents than the spurious cue, and it is plausible that [[claim:b2d08722-0692-48b5-9c00-b3628ceb99b5|spurious cues are just as well represented]]. A controlled confounded-label benchmark with an ideal or synthetic dictionary would settle it.",
  "confidence": 0.6,
  "claim_credence": 0.3,
  "marginal_yield": 0.35,
  "reasoning_trace": "What would make the claim true: the label is a simple function of a few latents, spurious cues are spread across many directions or held in latents a sparse selection skips, and dense probes are not regularized well enough to ignore them. What the evidence shows.\n(1) GDM progress update (Smith, Rajamanoharan, Conmy et al., March 2025): train on HarmBench vs Alpaca with AIM, roleplay, GCG and cocky-genius jailbreaks and two benign transformations; OOD test uses held-out jailbreaks (evil confidant, Pliny), held-out benign transformations (all caps, synonyms), OR-Bench, AdvBench and internal chat data. Dense probe AUROC 1.0 train and val, 0.999 OOD. k-sparse SAE probes on chat-finetuned Gemma Scope SAEs, latents chosen by mean difference or by an L1 sweep then refit unregularized, near-perfect in distribution for k of 5 or more but worse OOD; 1-sparse probes fail to fit training. Probes on SAE reconstructions lose 0.04 to 0.05 AUROC OOD, so the SAE drops task-relevant information. The dense probe's regularization and tuning are not described in the post. Spurious formatting cues (e.g. only Alpaca contained questions) were found with SAE latents and normalized away before training. The authors write that they are now unsure whether even an optimal SAE should beat linear probes OOD, and list four explanations, including that spurious patterns may be represented by latents as well as the true signal.\n(2) Kantamneni, Engels, Rajamanoharan, Tegmark, Nanda (arXiv 2502.16681, ICML 2025): 113 binary datasets, Gemma 2 9B layer 20, last token. Dense baseline L2 logistic regression with 10 regularization values chosen by validation AUC, plus PCA, KNN, XGBoost, MLP. SAE probes: top-k latents by mean activation difference, then L1 logistic regression. Covariate shift (8 OOD datasets, 300 test examples each: GLUE-X, translation, name and cartoon substitutions): logistic regression beats the SAE probe (width 131k, L0 114). Label noise up to 50%: no SAE advantage. Pruning latents o1 judged spurious improved OOD AUC by 0.024 and 0.052 on two tasks and hurt a third; the authors conclude spurious latents are not the main cause, rather latents that fail to generalize (an English-only 'living room' latent). Non-SAE remedies (max-activating examples of a linear probe) found the same dataset issues the SAE did.\n(3) Neither study induces a controlled spurious correlation. Marks et al.'s SHIFT and SAEBench's SCR metric do use a confounded Bias in Bios set, and show that ablating gender latents can improve a dense probe's worst-group accuracy; that is a different remedy (human or automated ablation), not evidence that sparse probes avoid spurious cues by default, so it is related rather than contrary evidence.\nWeighing: the predicted advantage did not appear in the two most careful tests, and the mechanism is undermined by evidence that SAE latents capture surface cues readily. The 'sufficiently good' qualifier and the lack of a controlled confound test keep this at doubtful rather than contradicted. Credence 0.3 for the claim as read today about practically achievable SAEs."
}