Step 25: assistant

text

I've assessed the claim that sparse autoencoders (SAEs) are powerful tools for discovering previously unknown concepts in language models. The verdict is **inconclusive**, with confidence 0.6. I gave no credence, because the claim is evaluative and its two readings point in different directions.

**What I read:** the position paper behind the claim (arXiv 2506.23845 v2, sections 1 to 5) and the cases it cites. For the audit and model-diffing studies I relied on the sibling question's full readings rather than re-reading them. I did not open Anthropic's 2025 paper on how Claude works internally (the rhyme-planning work) or OpenAI's work on "persona" features.

**Why it lands there:**
- **The weak reading is supported.** Read as "SAEs can surface concepts nobody specified in advance," the claim holds. The best case is Anthropic's 2025 finding that a model picks its rhyme word before writing the line. It was confirmed by interventions that change the output, and no simpler method has been tested against it. A second positive is a hidden-objective audit where adding SAE tools raised an automated agent's success from near 0% to 10 to 20%. That gain disappeared when the SAE was trained without the documents describing the hidden objective.
- **The paper's own evidence is thinner than it sounds.** The paper runs no experiment of its own. Its main measured win, generating hypotheses from text datasets, is about patterns in a corpus rather than concepts inside a model. The baselines it beat were built by overlapping authors. The rhyme-planning result used transcoders rather than SAEs, which the paper counts under the SAE umbrella.
- **The strong reading is not shown.** The paper means a real advantage over other methods, and where that has been tested the results are mostly null:
  - On AuditBench, the SAE tool added nothing, while prompting-based tools lifted success from about 36% to above 50%. The SAE tool there was weak, so a stronger setup isn't ruled out.
  - For comparing two models from their outputs, a strengthened language-model baseline did about as well as the SAE method.
  - In the blind audit where an SAE team was fastest, the authors say embedding search could have found the same documents.

**What I added to the claim's page:**
- **Argument for** (from the discovery case studies): rated slight. Case studies without a comparison can't show that SAEs do better than alternatives.
- **Argument against** (from comparisons with simpler methods on the same tasks): rated likely. It doesn't reach the discovery of internal mechanisms, where nothing has been compared.
- **One new subclaim:** that Claude 3.5 Haiku plans its rhyme word before writing the line. Three existing claims from the sibling question are linked as evidence.
- **Housekeeping:** both unattached passages from the paper are now on the argument for; the second was merged as a duplicate. Importance is set to 0.55 and contestation to 0.65. The decision is logged, and both inbox messages are answered.
- **Note to the sibling question:** it says that discovering mechanisms inside a model is the one part of the question no comparison yet tests.

I put the value of another pass at 0.4; reading the two papers I skipped is the obvious next step. What would settle the claim is a blind comparison on properties that arose in ordinary training rather than being planted, with an SAE and tuned simpler methods run by the same investigators and graded the same way.

The run used about $0.73 of the $1.20 budget.