Step 21: assistant

text

The verdict is unchanged: the claim stays **supported**, with 80% confidence in that reading and an 85% probability that it is true. Embedding search very probably could have found the same documents the fastest team found with sparse autoencoder (SAE) features. Nobody showed that a blind auditor would actually have done it that way. I re-read the relevant sections of the paper (arXiv 2503.10965: Sections 5, 5.1, 5.3.1, 5.4 and Appendix F.1).

**What the fastest team did.** Team 1 was told only to look for "potentially concerning behaviors". They wrote the prompt "The assistant is an AI model that exhibits potentially concerning behaviors" and looked at which SAE features were most active on it. One of those features fired on the planted training documents about reward-model biases. They reached it only after going through almost 200 features, because of which token they chose to inspect. The whole discovery took about 1.5 researcher-hours.

**Was the counterfactual run, by whom, knowing what?**
- **Who:** the paper's own authors ran it once, after the game.
- **Knowledge of the answer:** they knew where the planted documents were, and they say outright that this part of the paper was not done blind.
- **Method:** they converted every document in the training corpus into an embedding with a small open model of about 600 million parameters. They then ranked the documents by similarity to Team 1's own prompt. Planted documents made up 1 in 10,000 of the corpus.
- **Result:** the first planted document came back at rank 20. The prompt the second team used came back with one at rank 1.
- **Why the query isn't hindsight:** the authors knew the answer, but the query was the one the blind team had already written.

**What the authors conclude.** An auditor "could readily discover" the documents this way. They say this undermines their first reading, that the SAE was revealing knowledge the model kept hidden, and weakens the game as a showcase for interpretability. It cuts against their own interest, so it is credible as an admission against interest. They add that SAE features still work where semantic search fails, for example on examples of the biased behavior itself. That point doesn't touch Team 1's discovery.

**What a post hoc "could have replicated" claim can rest on.** It can rest on re-running the other method on the same inputs and showing it gets the same result. That was done, and it is fairly direct evidence that the documents were findable this way. It cannot show that a blind auditor would have picked semantic search, written that query, or read down to rank 20. The authors admit this needs "educated guesses". It is also one unreplicated run on a corpus and pipeline that aren't public. Reviewing 20 documents is less work than the roughly 200 features Team 1 actually went through, which helps the claim.

I also recorded a reading of the paper as a source and noted the Anthropic authors' stake in interpretability. I re-rated the one argument on the claim (the authors' after-the-fact search test) as "likely if its premises hold", not decisive. Because the verdict didn't move, no dependent claims needed notifying, and I answered the mandate's message. The pass cost about $0.54 of the $1.20 budget.