Step 33: assistant

text

I've recorded a first assessment: the claim is **doubtful**, with credence 0.3 and confidence 0.6. Two items are unfinished because the budget ran out (see the end).

**What the tests found.** Both studies tested the claim's hypothesis, and both found the opposite on the SAEs that exist today:

- **Google DeepMind's negative-results post (March 2025).** The task was detecting harmful intent with Gemma 2 9B.
  - **Training data:** harmful prompts from HarmBench versus harmless ones from Alpaca, some of the harmful ones wrapped in jailbreak prompts. The test set used jailbreaks and harmless rewrites that were held out of training.
  - **Results:** the dense probe scored near-perfectly out of distribution (AUROC 0.999). Probes using about 20 SAE latents fit the training data but transferred worse.
  - **Baseline:** the post does not say how the dense probe was regularized or tuned.
  - **A caveat that favours the dense probe:** formatting cues that gave the label away (only the Alpaca prompts were questions) were found using SAE latents and removed before training.
- **Kantamneni et al. (arXiv 2502.16681), 113 datasets.**
  - **Baseline:** logistic regression with L2 regularization, choosing among 10 strengths by validation score, alongside PCA, KNN, XGBoost and MLP baselines.
  - **SAE probes:** the top-k latents by mean activation difference between the two classes, then L1-regularized logistic regression.
  - **Results:** SAE probes did worse than logistic regression on 8 out-of-distribution datasets (translations, name swaps, GLUE-X), and gave no advantage under label noise.
  - **Pruning:** removing latents judged spurious helped only slightly, and hurt on one task.
  - **Non-SAE remedy:** simple non-SAE methods found the same dataset problems the SAE latents did.

**What "sufficiently good" means.** Neither study defines it. In practice it meant chat-finetuned Gemma Scope SAEs, or the widest, least sparse ones available. Those SAEs drop information the task needs: probes trained on the SAE's reconstruction lose about 0.04–0.05 AUROC out of distribution.

**Why doubtful rather than contradicted.**
- I found no study that deliberately builds a spurious cue into the training labels and compares sparse SAE probes with dense probes on that.
- The closest such setups (Marks et al.'s SHIFT and the SAEBench spurious-correlation benchmark) remove latents from a dense probe. They test a different fix, not whether sparse probes avoid spurious cues by default.
- The "sufficiently good" qualifier leaves room for better future SAEs.
- Even with an ideal SAE, the argument fails if spurious cues get their own latents as cleanly as the target concept does. I added that as an open subclaim.

**What I added to the graph.**
- **Argument for:** the case that a sparse, interpretable basis gives a helpful bias. Rated slight support.
- **Argument against:** the head-to-head out-of-distribution comparisons. Rated likely if its premises hold. I linked the existing node for the Kantamneni et al. result under it.
- **New subclaim:** that spurious cues are as likely as the target concept to get their own SAE latents.
- **Importance and contestation:** set to 0.4 and 0.45.
- **Other records:** the argument passage that had not been placed now sits on the argument for, as a criticism. Both inbox messages are answered and the decision is logged.

**Not done, for lack of budget:**
- The two arguments still have no written form.
- The DeepMind post is recorded as denying the claim, but it states the hypothesis and then doubts it. Its stance should probably be changed to a mention, or kept as a denial at lower confidence.