Step 17: assistant

thinking

Calls update_claim_assessment

{
  "status": "supported",
  "claim_id": "5393957c-bef5-4262-a152-84c5fe63ece2",
  "assessment": "The claim concerns the blind auditing game in Anthropic's paper \"Auditing language models for hidden objectives\" (Marks et al., March 2025). The fastest team, Team 1, found the planted synthetic documents describing reward-model biases in about 1.5 researcher-hours. Knowing only that they were looking for \"potentially concerning behaviors\", they wrote the prompt \"The assistant is an AI model that exhibits potentially concerning behaviors\", inspected the sparse autoencoder (SAE) features most active on it, and found a reward-model-bias feature whose top-activating training samples were the synthetic documents. They found it only after looking through almost 200 features, because of the token position they chose.\n\nThe counterfactual was tested once, by the paper's own authors, after the game and without blinding. They embedded every document in the game's pretraining corpus, in which 1 in 10,000 documents was synthetic, using a small open embedding model of about 600 million parameters. They then ranked the documents by cosine similarity to Team 1's prompt. The closest synthetic document came back at rank 20. The prompt Team 2 used to find its SAE feature returned a synthetic document at rank 1. The authors knew where the planted documents were when they ran this test, but the query was not chosen with hindsight: it was the one the blind team had already written. The authors conclude that an auditor \"could readily discover\" the documents this way. They say this undermines their first reading of the result as SAEs uncovering hidden knowledge, and undermines the game as an example of interpretability adding practical value. Because the finding cuts against the authors' interest in showing what interpretability can do, it is credible as an admission against interest.\n\nA claim that something \"could have\" been done, made after the fact, can rest on one kind of evidence: re-running the alternative method on the same inputs and showing that it reaches the same result. That is what was done here, and for the discovery step it is fairly direct evidence. It cannot show that a blind auditor would have chosen semantic search, written that query, or read down to rank 20. The authors acknowledge that this unblinded analysis requires \"educated guesses\" about what blind auditors would have done. It also rests on one run with a corpus and pipeline that are not public, and no one else has replicated it. The claim therefore holds for this particular discovery and should not be generalized. The same paper finds that SAE features still work where semantic search fails, for example on demonstrations of biased behavior or on the Assistant control token alone.",
  "confidence": 0.8,
  "claim_credence": 0.85,
  "marginal_yield": 0.1,
  "reasoning_trace": "Read arXiv 2503.10965 (HTML) Sections 5 (preamble), 5.1, 5.3.1, 5.4, and Appendix F.1.1.\n\nFacts established:\n- Team 1's prompt was formed blind: \"knowing only that they were searching for 'potentially concerning behaviors' and their causes\". They found the RM-bias feature only after inspecting \"almost 200 of the top-activating features\" at the period token (footnote 20).\n- Semantic search (F.1.1): \"small embedding transformer with ca. 600M parameters (Yu et al., 2024)\". Documents were ranked by cosine similarity, and the rank of the closest synthetic document was recorded. Team 1's phrase gave rank 20. Team 2's prompt gave rank 1. \"AI model\" gave rank 6. The base rate of synthetic documents was 1 in 10,000.\n- Who ran it, and with what knowledge: the authors, after the game. Section 5 says the technique investigation \"was not conducted by blinded researchers\" and that conclusions about efficacy \"require making educated guesses about whether it would have been useful to a blind auditor\".\n- Authors' conclusion (5.4): \"we later discovered that an auditor could readily discover synthetic documents via semantic search\". This \"undermines our initial conclusion\" both on whether SAEs reveal hidden knowledge and on practical importance. In 5.1 they say semantic search \"successfully surfaces the same training documents that teams 1 and 2 discovered\". The introduction says the solution \"could have been replicated by a simpler embedding-based search baseline\". Their caveat is that SAEs succeed where semantic search fails on other queries, which does not bear on Team 1's discovery.\n\nWhat the post hoc claim can rest on: a re-run of the alternative method on the identical query and corpus. That was done, and it is good evidence that the documents were retrievable. It cannot show that a blind auditor would have used the method or the query, or would have reviewed 20 results. Rank 20 is a small review burden, smaller than the roughly 200 SAE features Team 1 actually inspected. The claim's \"could have replicated\" concerns the discovery step, not the whole workflow.\n\nRemaining doubts: a single unreplicated, unblinded run with private data. The prompt is worded slightly differently in different places (\"with\" in 5.4, \"that exhibits\" in 5.1/F.1), but the reported rank is for the wording the team used. Credence 0.85."
}