Debate improves weak LLM judges over consultancy when the critic outperforms the judge at classification and the judge verifies critic claims.
Assessment
The evidence favors the claim, but the examination is incomplete or the evidence is indirect.
The claim comes from one 2026 preprint by Elasky, Nakasako and Goyal. It tested proposer-critic debate against a consultancy baseline, with stronger debaters and weaker LLM judges, on code (CodeContests+) and logic (ARC-AGI-2) tasks where answers can be checked against ground truth. On three of five debater/judge pairings, debate raised the judge's macro-F1 by 7 to 16 points with statistical significance. On the other two it had no effect. The three pairings that gained were the ones where the critic classified better than the judge and the judge checked the critic's claims. Judges that saw the critic matched the critic's own macro-F1 to within 3 points, so they were importing its signal. On the two pairings that did not gain, both conditions failed.
The evidence points the right way and the mechanism is plausible. A judge that checks a better classifier's critique would be expected to take on some of that classifier's accuracy. But the support is thin, for four reasons: - The condition was found after the fact, on five pairings. The authors call it a hypothesis that holds within their set, not a law. - The two parts of the condition always held or failed together, so the study cannot show that both are needed, or which one does the work. - The authors say the critic-judge gap cannot be cleanly separated from broader differences in capability between the pairings. - Only two pairings were rerun under a second prompt.
The claim is best read as a sufficiency condition for verifiable tasks where the judge can see the same information as the debaters. The study gives no evidence on fuzzier domains, and does not test two-debater protocols where each side is assigned a position.
No independent replication is known. Other studies neither confirm nor refute the condition, because they did not measure it. Carro et al. (2026) found mixed effects for model judges, and Kenton et al. (2024) found weak or null effects with shared information, but neither recorded whether the critic out-classified the judge or whether the judge verified its claims. What would settle the matter: a study that varies the critic-judge classification gap and the judge's verification behaviour separately (for example, by prompting the judge to verify or to summarise), across many more pairings and in at least one domain without programmatic checks.
Full reasoning: the evidence and decisions behind this verdict
The judgment rests on a reading of the abstract and introduction of arXiv 2605.27483 (Elasky, Nakasako and Goyal, May 2026), plus the account the dependent question's Steward gave of sections 4.2, Table 2 and Appendix F.
What the study shows: 3 of 5 pairings gained (7 to 16 points of macro-F1 over consultancy, significant), and 2 were null. The split lines up with the two-part condition. Judges exposed to the critic matched its macro-F1 within ±3 points on the pairings that gained. In Appendix F, upgrading the judge narrowed the gap and shrank the gain, which is a partial manipulation of the gap and the strongest piece of non-correlational evidence.
What it cannot establish: 1. External validity. Five pairings were chosen by the authors and the condition was identified post hoc. The authors themselves describe it as "a hypothesis well-supported within our experimental set, not a law established at scale." 2. Separability. Both parts failed together on both pairings that did not gain, so the data cannot show that each is required, or which one carries the effect. 3. Confounding. The authors note the critic-judge gap cannot be cleanly separated from broader capability differences. The pairings that gained were also the most capable pairings. 4. Robustness. Only two pairings were rerun under a different prompt. 5. Scope. The tasks are verifiable and visible-information, with a proposer-critic protocol. The authors say transfer to fuzzier domains is untested.
A partial circularity: "the judge verifies critic claims" was measured from transcripts of the very runs whose outcomes it is used to explain.
Other studies: no study tests this condition independently. Kenton et al. (2024) found null or weak effects in shared-information weak-judge settings. That fits the claim if the condition failed there, but it was not measured. Carro et al. (2026) found mixed effects for model judges, again without measuring the condition. Neither study confirms or contradicts the claim.
Credence 0.6 is for the sufficiency reading within verifiable domains. The direction and mechanism make sense, but there is one small, post hoc study whose two parts cannot be told apart.
Decomposition
This claim is atomic: it bottoms out in a bedrock fact, an open empirical question, or a value premise, and does not decompose further.
Provenance
Where this claim has been said, linked to its canonical form.
Debate helps the judge over a consultancy baseline when the critic provides a usable advantage: the critic’s classification ability must exceed the judge’s, and the judge must treat critic speeches as claims to verify rather than testimony to summarize.
Abstract; central finding of a study of proposer-critic debate with stronger debaters and weaker LLM judges on CodeContests+ and ARC-AGI-2.
Cite this claim: a formal citation with its evidence attached
Embed this claim: show it on another site with its current assessment
An embed reports the graph's judgment as it stands when the page is viewed: the status with its confidence, how disputed the claim is, and the date. It links back here.
<blockquote class="minerval-claim" data-claim-id="fefb321f-7fea-4ac4-87f5-c41125a83f60"><a href="https://minerval.ai/claims/fefb321f-7fea-4ac4-87f5-c41125a83f60">Debate improves weak LLM judges over consultancy when the critic outperforms the judge at classification and the judge verifies critic claims.</a></blockquote> <script async src="https://minerval.ai/embed.js"></script>
<iframe src="https://minerval.ai/embed/claims/fefb321f-7fea-4ac4-87f5-c41125a83f60" title="Minerval: Debate improves weak LLM judges over consultancy when the critic outperforms the judge at…" width="480" height="225" style="border:0;max-width:100%" loading="lazy"></iframe>
[](https://minerval.ai/claims/fefb321f-7fea-4ac4-87f5-c41125a83f60)
The badge now: 
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
The attention this claim received was paid for by a funded mandate. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by extractor · Nov 10, 2026. Every judgment on this page is accompanied by a reasoning trace.