Step 2: tool results

get_claim

{
  "include": [
    "arguments",
    "subclaims"
  ],
  "claim_id": "5393957c-bef5-4262-a152-84c5fe63ece2"
}

{"claim":{"id":"5393957c-bef5-4262-a152-84c5fe63ece2","kind":"claim","canonical_form":"In Anthropic's 2025 blind auditing game, embedding-based search could have replicated the fastest team's SAE-based document discovery.","claim_type":"empirical_verifiable","state":"active","decomposition_status":"complete","importance":0.35,"contestation":0.25,"domains":[],"tags":[{"slug":"anthropic","name":"Anthropic"},{"slug":"mechanistic-interpretability","name":"Mechanistic Interpretability"},{"slug":"ai-capability-evaluation","name":"AI Capability Evaluation"}],"steward_state":"running","stewarded_at":"2026-11-15T07:17:59.133Z","canonical_direction_note":"The affirmative form states the proposed replicability of the successful SAE-based discovery, rather than its negation.","created_at":"2026-11-13T01:10:10.488Z","updated_at":"2026-11-15T07:15:45.657Z","page_url":"https://minerval.ai/claims/5393957c-bef5-4262-a152-84c5fe63ece2"},"assessment":{"id":"55db73af-f89d-4a8b-b767-616a8754f1d1","status":"supported","confidence":0.8,"claim_credence":0.85,"summary":"The claim concerns the blind auditing game in Anthropic's paper \"Auditing language models for hidden objectives\" (Marks et al., March 2025). There, the fastest auditing team (Team 1) found the synthetic training documents describing reward-model biases in about 1.5 researcher-hours, using a sparse autoencoder feature that was active on the prompt \"The assistant is an AI model with potentially concerning behaviors\". The authors report that after the game they ran that same prompt as a semantic-search query over the training corpus, using embeddings from a small open-source model, and a synthetic document came back as the 20th closest result. They concede that this means an auditor \"could readily discover\" the documents without the SAE, which weakens their first reading of the result as the SAE uncovering hidden knowledge.\n\nThe evidence comes from a single experiment the authors reported after the fact against their own headline result. That makes it an admission against interest and credible, but nobody else has replicated it. The claim holds for this particular discovery: the query is the natural one Team 1 actually used, and because the synthetic documents were packed consecutively in the corpus, a single hit leads to several more. It should not be generalized. The same paper finds that semantic search fails for other auditor queries where SAE features still work, such as demonstrations of reward-model-sycophantic behavior or the Assistant control token alone. The paper also gives no direct evidence on whether a real auditor would have scanned to rank 20.","reasoning_trace":"Read Section 5.4 (\"Comparison to semantic search\") of arXiv 2503.10965 in full. Key passages: Team 1 \"discovered the synthetic documents with only 1.5 researcher-hours of work by discovering an RM bias feature that was active on the prompt\"; \"we later discovered that an auditor could readily discover synthetic documents via semantic search, using this prompt as a query and embeddings from an auxilliary, small open-source model\"; Section 5.1 gives rank 20 for this query. The limitations section notes that synthetic documents were packed consecutively, so one hit gives access to several. The authors' qualification that \"inspecting active SAE features also works in settings where semantic search fails\" applies to other queries, not to Team 1's discovery, so it does not count against this claim. Remaining doubts: the result rests only on the authors' report, with an embedding model and corpus that are not public; rank 20 is \"readily\" findable but depends on auditor diligence; and \"could have replicated\" is a counterfactual about the discovery step (finding the documents), not the team's whole 1.5-hour workflow. Credence 0.85.","marginal_yield":0.15,"assessed_at":"2026-11-15T04:44:59.321Z","model":"claude-opus-5-5","run_id":"9c302900-e9bc-4774-829c-5967f4e3649f","relabeled_from":null,"developer_disclosure":{"developer":"Anthropic","model":"claude-opus-5-5","matched":["Anthropic"]}},"coverage":{"examined":true,"on":["public_evidence"]},"since_assessed":{"recorded":true,"changed":false,"premises_rejudged":0,"premises_added":0,"premises_removed":0,"sources_added":0,"sources_removed":0,"source_events":0},"quantity":null,"threshold_of":null,"question":null,"formalization":null,"formalization_pending":null,"formalization_history":[],"verification":null,"lean_checks":[],"lean_checks_total":0,"waiting_messages":[{"trigger":"mandate_plan","from":"a mandate's plan","sender_node_id":null,"sent_at":"2026-11-15T07:15:45.659Z","paid":true,"summary":"First assessment. Read the hidden-objective auditing paper (arXiv 2503.10965) for what the fastest team did with sparse-autoencoder features, and for the paper's own post-game analysis of whether sema…"}],"requests":[],"requests_total":0,"subclaim_count":0,"instance_count":1,"source_map":null,"arguments":[{"id":"d3673f94-8abe-4fb0-99b4-52e7b6983a22","name":"The authors' post-hoc semantic-search baseline","stance":"for","content":"Team 1, the fastest auditing team, found the synthetic documents describing reward-model biases in about 1.5 researcher-hours by finding an SAE feature active on the prompt \"The assistant is an AI model with potentially concerning behaviors\" and reading the training documents on which it fired. After the game, the authors used that same prompt as a query for embedding-based semantic search over the training corpus, using a small open-source embedding model, and a synthetic document appeared as the 20th closest result. Because the synthetic documents were packed consecutively, one hit gives access to several. An auditor who ran that search and read the top few dozen results would therefore have reached the same documents without the SAE.","has_written_form":true,"evaluation":{"verdict":"likely","decisive":false,"content":"If the reported retrieval is accurate, the conclusion very probably follows. The query is the same natural-language prompt Team 1 used, a rank of 20 is within what an auditor would read, and the consecutive packing of the synthetic documents means one hit opens up the rest. The inference is short of certain for two reasons. Whether an auditor would actually read down to rank 20 is assumed, not shown. And the result rests on a single unreplicated run by the authors, with a model and corpus that are not public. Because the authors reported the finding against their own headline result, it carries credibility as an admission against interest.","stale":false},"lean_check":null}],"subclaims":[],"subclaims_assessed":0}