Weaker AI judges gain accuracy from debate comparably to human judges.
Assessed
First pass on a tight budget. The claim is atomic; no subclaims or named arguments, since the evidence is a few direct comparisons weighed in the assessment. Recorded Khan 2024 (affirms, with information asymmetry) and Carro 2026 (denies, belief correction). Assessed doubtful with credence 0.3. Importance 0.4, contestation 0.5. Not done: a close reading of Rahman et al. 2025.
Assessed Doubtful
verdict confidence 0.65 · credence 0.30
Stated generally, the claim that weaker AI judges gain as much accuracy from debate as human judges do is not borne out by the few studies that put both kinds of judge through the same protocol. Whether the gains match depends on the setting. Where debaters can see a source text the judge cannot, the gains have matched. In a 2024 reading-comprehension study (Khan et al.), debate raised model judges from 48% to 76% and human judges from 60% to 88%, about 28 points each. Human judges were still more accurate in absolute terms under every protocol. Model-only work from the same period (Kenton et al., 2024) also found that weak model judges benefit from debate on hidden-text tasks, but got mixed results on tasks without that information gap. Where the task is correcting judges' own prior beliefs, with no hidden text, the gains have not matched. In a 2026 study on claims about scientific discoveries (Carro et al.), debate raised human judges from 41% to 58%. Model judges did worse: a few proprietary models gained less than humans, others showed no gain, and small open-weight models lost accuracy. Voting and deliberation among several model judges did not help either. That comparison is not exactly like for like, since human priors were elicited while model priors were induced by prompt. The evidence therefore supports comparable gains only under information asymmetry, and only on one task family. There it is not shown to hold in general, and in the one belief-correction comparison it fails. More head-to-head studies across task types and judge strengths would settle the question.
Claim entered the graph