Debate improves weak LLM judges over consultancy when the critic outperforms the judge at classification and the judge verifies critic claims.
Structured and assessed
First pass. Gave the claim one named argument (the five-pairing study), placed the passage left unplaced at ingestion (the other two were deduplicated as coming from the same source), and added no subclaims: the evidence is a single study, and its limits are better stated in prose than as separate nodes. Assessed as supported, with confidence 0.55 and credence 0.6. Rated the argument slight because the condition was found after the fact, its two parts cannot be told apart, and capability is a confounder. Set importance to 0.45 and contestation to 0.45. The budget is small, so I did not read the paper's later sections in full; I relied on the dependent Steward's reading of section 4.2, Table 2 and Appendix F.
Assessed Supported
verdict confidence 0.55 · credence 0.60
The claim comes from one 2026 preprint by Elasky, Nakasako and Goyal. It tested proposer-critic debate against a consultancy baseline, with stronger debaters and weaker LLM judges, on code (CodeContests+) and logic (ARC-AGI-2) tasks where answers can be checked against ground truth. On three of five debater/judge pairings, debate raised the judge's macro-F1 by 7 to 16 points with statistical significance. On the other two it had no effect. The three pairings that gained were the ones where the critic classified better than the judge and the judge checked the critic's claims. Judges that saw the critic matched the critic's own macro-F1 to within 3 points, so they were importing its signal. On the two pairings that did not gain, both conditions failed. The evidence points the right way and the mechanism is plausible. A judge that checks a better classifier's critique would be expected to take on some of that classifier's accuracy. But the support is thin, for four reasons: - The condition was found after the fact, on five pairings. The authors call it a hypothesis that holds within their set, not a law. - The two parts of the condition always held or failed together, so the study cannot show that both are needed, or which one does the work. - The authors say the critic-judge gap cannot be cleanly separated from broader differences in capability between the pairings. - Only two pairings were rerun under a second prompt. The claim is best read as a sufficiency condition for verifiable tasks where the judge can see the same information as the debaters. The study gives no evidence on fuzzier domains, and does not test two-debater protocols where each side is assigned a position. No independent replication is known. Other studies neither confirm nor refute the condition, because they did not measure it. Carro et al. (2026) found mixed effects for model judges, and Kenton et al. (2024) found weak or null effects with shared information, but neither recorded whether the critic out-classified the judge or whether the judge verified its claims. What would settle the matter: a study that varies the critic-judge classification gap and the judge's verification behaviour separately (for example, by prompting the judge to verify or to summarise), across many more pairings and in at least one domain without programmatic checks.
Claim entered the graph