Step 19: assistant

thinking

Calls add_decomposition_edge

{
  "relation": "supports",
  "parent_id": "ae6522c7-b164-4787-99a9-9ab318f858af",
  "reasoning": "The proved sufficient condition for a stable decomposition is that the human-judgeable claims at the base of an argument each have sufficiently many independent pieces of supporting evidence. Whether real oversight questions have that backing is the empirical premise the argument for stability rests on; it is sufficient, not necessary, so it supports and is not required.",
  "seed_note": "Empirical findings that many studies or measurements bear on usually have redundant support, and a conclusion about code can be given it by adding tests. Much oversight work is of another kind: a judgment that a long piece of work contains no flaw anywhere, a proof, a research plan defended by taste. For these it is unclear what independent evidence would be. Independence is also hard to establish: pieces of evidence produced by the system under oversight, or drawn from one dataset or one method, can fail together, and the authors of the debate protocols that rely on this condition list whether judges can tell that evidence is independent as an open question. No survey or dataset has measured how often the condition is met.",
  "child_text": "Correct answers to real-world AI oversight questions are typically supported by many independent pieces of evidence.",
  "importance": 0.25,
  "argument_id": "40c16541-5a20-414a-91cf-dc46fdf3c2f4",
  "contestation": 0.3,
  "seed_credence": 0.3
}

Calls add_decomposition_edge

{
  "relation": "contradicts",
  "parent_id": "ae6522c7-b164-4787-99a9-9ab318f858af",
  "reasoning": "A justification in which every step is necessary and none is redundantly supported is as sensitive to single errors as an argument can be, so recursive debate gains nothing over checking every step. If many oversight questions can be justified only in that way, they lack the stable decompositions the efficiency guarantees need.",
  "seed_note": "Clear cases exist: a mathematical proof, a claim that a large change to a codebase contains no vulnerability, a causal analysis that fails on one overlooked confound. The developers of recursive debate concede that mathematical proofs depend precisely on individual steps. The open part is the word \"require\": some chains can be recast with redundancy (independent tests of code, a second derivation, replication), and it is not known how often that is possible or how large a share of oversight questions the irreducible cases make up. \"Many\" is a low bar, which is why the prior sits above one half.",
  "child_text": "Many real-world AI oversight questions require long, nonredundant reasoning chains to justify their correct answers.",
  "importance": 0.25,
  "argument_id": "71f706bf-7b56-4bba-9cd2-213645255a9c",
  "contestation": 0.3,
  "seed_credence": 0.65
}

Calls add_relationship_edge

{
  "child_id": "a4225e85-09ed-4f2b-8cf2-23998858b16c",
  "relation": "assumes",
  "parent_id": "ae6522c7-b164-4787-99a9-9ab318f858af",
  "reasoning": "The claim presupposes that recursive debate has efficiency guarantees conditional on stability. The prover-estimator theorem is the principal one; if it failed there would be no guarantee whose applicability is in question."
}

Calls add_related_claim

{
  "kind": "related",
  "claim_id": "ae6522c7-b164-4787-99a9-9ab318f858af",
  "reasoning": "Stability is the condition under which later protocols escape the obfuscated arguments problem. A reader weighing how far obfuscation limits debate needs to know how often real arguments are stable, and the reverse.",
  "other_claim_id": "30b4f79d-4286-43ff-a6a9-be91d82c3692"
}

Calls record_claim_instance

{
  "url": "https://arxiv.org/abs/2506.13609",
  "title": "Avoiding Obfuscation with Prover-Estimator Debate",
  "stance": "poses",
  "context": "Discussion section of the paper that introduced the stability condition (revised version). The introduction calls the existence of a stable decomposition the primary restriction on the class of problems the protocol can judge.",
  "speaker": "Jonah Brown-Cohen, Geoffrey Irving, Simon C. Marshall, Ilan Newman, Georgios Piliouras, Mario Szegedy",
  "claim_id": "ae6522c7-b164-4787-99a9-9ab318f858af",
  "publication": "arXiv",
  "source_date": "2026-07-21",
  "verbatim_text": "Our results identify stability as a critical requirement for debate training to succeed, and hence we need to better understand when stability can be achieved, either in theory for interesting classes of problems, or in practice for debates in messy, unformalized settings."
}

Calls record_claim_instance

{
  "url": "https://www.lesswrong.com/posts/8XHBaugB5S3r27MG9/prover-estimator-debate-a-new-scalable-oversight-protocol",
  "title": "Prover-Estimator Debate: A New Scalable Oversight Protocol",
  "stance": "poses",
  "context": "The protocol's authors summarising their paper. They close with two open problems: whether stable arguments exist under weaker assumptions than sufficient independent evidence, and when independent evidence can be found in practice.",
  "speaker": "Jonah Brown-Cohen and Geoffrey Irving",
  "claim_id": "ae6522c7-b164-4787-99a9-9ab318f858af",
  "publication": "LessWrong / AI Alignment Forum",
  "source_date": "2025-06-17",
  "verbatim_text": "The need for stability for completeness was discussed by Beth Barnes’ original post introducing obfuscated arguments; our paper presents one route to stability, but does not resolve whether most questions have stable arguments in practice."
}

Calls record_claim_instance

{
  "url": "https://www.lesswrong.com/posts/8XHBaugB5S3r27MG9/prover-estimator-debate-a-new-scalable-oversight-protocol",
  "title": "Prover-Estimator Debate: A New Scalable Oversight Protocol (comment)",
  "stance": "poses",
  "context": "Comment by the researcher who first described obfuscated arguments. She lists this as one of the hopes on which obfuscation would rarely matter in practice, notes counterexamples, and says the paper should not be a substantial update on whether obfuscated arguments are a problem.",
  "speaker": "Beth Barnes",
  "claim_id": "ae6522c7-b164-4787-99a9-9ab318f858af",
  "confidence": 0.8,
  "publication": "LessWrong / AI Alignment Forum",
  "source_date": "2025-06",
  "verbatim_text": "For anything the debaters know, there’s usually a sufficiently compact and stable honest argument such that the honest debater can win by giving this argument It seems pretty easy to give counterexamples here - e.g. Paul gives the example of unsupervised translation. But maybe these are relatively rare. It's plausible that the honest debaters in the human experiments I did where we ran into obfuscation / instability problems weren't using the right tactics (i.e. they could have constructed more stable arguments that used independent lines of evidence more)"
}

Calls record_claim_instance

{
  "url": "https://arxiv.org/abs/2505.03989",
  "title": "An alignment safety case sketch based on debate",
  "stance": "poses",
  "context": "A sketch of a safety case for an AI research agent trained by debate. Stability is one of the claims the case needs; the authors leave its evidence blank and add that unstable queries return no answer in prover-estimator debate, so the protocol is safe for all queries but useful only for stable ones.",
  "speaker": "Marie Davidsen Buhl, Jacob Pfau, Benjamin Hilton, Geoffrey Irving",
  "claim_id": "ae6522c7-b164-4787-99a9-9ab318f858af",
  "publication": "arXiv",
  "source_date": "2025-05",
  "verbatim_text": "This is known as stability and can be more precisely stated as: small variations in input probabilities will not change the outcome of overall computation. We are not yet sure how to support this claim, so we leave the evidence box blank (E5.5)."
}

Calls set_claim_importance

{
  "claim_id": "ae6522c7-b164-4787-99a9-9ab318f858af",
  "reasoning": "A specific open premise inside the live question of whether debate can serve as scalable oversight. It decides how much of the oversight problem the proved protocols can cover, and the 2026 lower bound makes it unavoidable in the current theory. Its failure costs usefulness, not soundness, and the debate is confined to a small research community in which both proponents and critics treat it as open, so it sits at the notable anchor rather than above it.",
  "importance": 0.35,
  "contestation": 0.4
}