Obfuscated arguments limit debate and iterated amplification to verifying arguments small relative to training computation.
Assessed
First pass, with about half the budget already consumed at the start. Recorded the claim as inconclusive (credence 0.4) and left it atomic, with no named arguments: the crux is whether the stability/decomposability assumptions of obfuscation-resistant protocols hold for real arguments. This does not warrant new nodes at this importance. Instances were not recorded for lack of budget; a later pass should add Barnes & Christiano 2020 and the prover-estimator debate paper.
Assessed Inconclusive
verdict confidence 0.65 · credence 0.40
The obfuscated arguments problem was described by Beth Barnes and Paul Christiano in a December 2020 report on OpenAI's human debate experiments. A dishonest debater could present a long argument containing a single fatal flaw while the honest debater, though confident the argument was wrong somewhere, could not locate the flaw with the queries available. Because recursive decomposition is the core of both debate and iterated amplification, the authors suggested this might confine both schemes to verifying arguments whose size is small compared with the computation spent training the system. They presented this as a likely limitation and admitted they had no fix. They did not establish it as a theorem. The phenomenon is well documented. Whether it amounts to a hard limit is unsettled. The evidence for it comes from small-scale human experiments and complexity-style reasoning: finding a flaw in an argument that hides one among many plausible steps can be as hard as the whole problem. Against it, later theoretical work shows that modified debate protocols avoid obfuscation under stated assumptions. Doubly-efficient debate (Brown-Cohen, Irving and Piliouras, 2023) assumes access to a verifiable decomposition of the problem. Prover-estimator debate (2025) assumes the argument is stable, meaning its conclusion does not hinge on tiny changes in the probabilities of its steps. Whether real-world arguments satisfy those assumptions, and whether trained models can carry out the protocols, has not been tested empirically at scale. The extension from debate to amplification also rests on the shared decomposition structure, not on direct experiments. The claim reads as a definite limit, and on present evidence it is a serious, credible conjecture that is neither confirmed nor refuted. It would be resolved by either of two results: empirical tests of obfuscation-resistant protocols with trained models on long arguments, or impossibility results for the kinds of argument that fail the stability conditions.
Claim entered the graph