Step 26: tool results

{
  "ask": "Measure how sensitive the best available decompositions of realistic AI oversight questions are. Take a sample of questions of the kind debate is meant to judge (review of a substantial code change for correctness and for hidden flaws, evaluation of an empirical research write-up, evaluation of a research plan), have capable honest arguers, human or model, construct the most stable decomposition they can for the true answer down to leaves a human can judge in minutes, and report for each question the per-step fractional block sensitivity (or the equivalent chance that sampling one subclaim catches a single planted error), the number of subclaims per step, the depth, and whether the redundancy used rests on evidence that is demonstrably independent.",
  "why": "Whether real oversight questions admit decompositions stable enough for recursive debate's efficiency guarantees is unmeasured, and the present assessment (doubtful, credence about 0.25 for the general claim) rests on structural argument and on the protocol developers' own statements that the matter is open. If most sampled questions, including flaw-finding ones, have decompositions with sensitivity far below the number of subclaims at workable depth, the claim would move to supported. If stable decompositions are found only for evidence-pooling questions and not for flaw-finding or taste-dependent ones, the mixed picture would be confirmed and the general claim would stay doubtful with much higher confidence. If honest arguers with good tactics cannot find stable decompositions for most questions, it would move to contradicted.",
  "claim_ids": [
    "ae6522c7-b164-4787-99a9-9ab318f858af"
  ],
  "who_could": "A group working on debate protocols with access to expert human debaters or strong models, such as the UK AI Security Institute alignment team or academic collaborators. A desk stage, searching for any published dataset or experiment of this kind after October 2026, is within a Researcher run.",
  "satisfied_by": "A table with one row per question giving the task type, depth, subclaims per step, measured or computed sensitivity per step, and the basis for treating supporting evidence as independent, with the method used to search for the most stable decomposition. A documented finding that such a study has already been published, with its figures, also satisfies it. Any pattern of results satisfies it."
}

{
  "claim_id": "ae6522c7-b164-4787-99a9-9ab318f858af",
  "reasoning": "First pass. Read the claim as a general present-tense statement that real oversight questions admit decompositions with sensitivity low enough for the proved efficiency guarantees. Formed a prior of 0.2 to 0.3 from the structure of the condition (all-parts-necessary conclusions have maximal fractional block sensitivity; thresholds with margin have constant sensitivity). Read directly from arXiv sources: the introduction and discussion of the revised prover-estimator paper (2506.13609), the introduction and discussion of the October 2026 instance-optimal paper (2610.02557), key claim 2 of the debate safety-case sketch (2505.03989), and the authors' forum post with Barnes's, Irving's and Pfau's comments. Web searches found no empirical measurement of stability and no dataset. Structure: three named arguments (redundant independent evidence, for; conclusions in which every step is necessary, against; instability in the 2020 human experiments, against). Minted two subclaims after the Matcher judged both novel: that correct answers to oversight questions typically have many independent supports (supports, seed 0.3) and that many oversight questions require long nonredundant chains (contradicts, seed 0.65), each at importance 0.25. Linked the prover-estimator theorem as an assumed framework premise and the obfuscated-arguments limitation claim as a see-also. Recorded four instances, all posing the claim as open; no source asserts or denies it. Wrote a provenance summary noting that the voices are one small research community. Assessed doubtful, confidence 0.55, credence 0.25, marginal yield 0.25. Importance set to 0.35 and contestation to 0.4 (notable premise in a live but narrow debate; failure costs usefulness, not soundness). Filed a request for a measurement of decomposition sensitivity on realistic oversight questions. No notification sent to the dependent claim on debate's assumptions: its assessment already took stability at about 0.3 and unmeasured, which this verdict matches. Not read: Barnes's 2020 write-up and the proof of the independent-evidence theorem; stated as limits in the reasoning.",
  "action_taken": "structured_and_assessed"
}

{
  "answers": [
    {
      "note": "First pass done: three arguments, two new subclaims, the prover-estimator theorem linked as an assumed premise, four instances recorded, and an assessment of doubtful (confidence 0.55, credence 0.25). Stability on realistic oversight tasks is unmeasured; the structural argument and the developers' own statements weigh against the general claim.",
      "message": "m1",
      "outcome": "changed"
    },
    {
      "note": "Assessment recorded: doubtful, credence 0.25, with importance 0.35 and contestation 0.4. A request was filed for a measurement of decomposition sensitivity on realistic oversight questions, which is what would settle it.",
      "message": "m2",
      "outcome": "changed"
    }
  ]
}