Step 16: tool results

{
  "note": "The sentence summarizes a post-game, unblinded experiment reported in Sections 5.1, 5.4 and Appendix F.1: the authors embedded every pretraining-corpus document with a roughly 600M-parameter open embedding model and found that Team 1's own prompt retrieves a synthetic document at rank 20 by cosine similarity (Team 2's prompt gives rank 1). The authors knew where the planted documents were when they ran it, but the query was the one the blind team had already chosen, so the retrieval result itself does not depend on hindsight. Whether a blind auditor would have chosen semantic search and read to rank 20 is not tested, and the authors say so. The corpus and model are not public, so no one outside the authors has re-run it.",
  "support": "supports",
  "deployment": "Stated in the introduction as a qualification of the paper's own headline result that the fastest team succeeded with sparse autoencoders.",
  "instance_id": "fc5438d9-aafa-41b4-be50-98783ae07c75",
  "source_kind": "preprint",
  "source_read": true,
  "worth_reading": true,
  "worth_reading_reason": "Sections 5.1, 5.4 and Appendix F.1 hold the only evidence for the counterfactual and its caveats."
}

{
  "party": "Anthropic (the paper's authors)",
  "scope": "claim",
  "stake": "The authors are Anthropic researchers, several of whom work on sparse-autoencoder interpretability, which the auditing game was partly designed to showcase. A finding that a simple search baseline matched the SAE result runs against that interest.",
  "claim_id": "5393957c-bef5-4262-a152-84c5fe63ece2",
  "source_id": "bf2135ba-f5be-48c4-82eb-271c5bc3db71"
}