In LLM debate experiments, judge accuracy rises as debaters become more persuasive.
Assessed first pass
Kept the claim atomic, with no named arguments: its support is one line of evidence, the Elo-versus-accuracy results. Recorded the Khan 2024 and Kenton 2024 instances. Assessed it as supported with credence 0.7, scoped to hidden-information tasks, and noted that the closed tasks give weak results and that the persuasiveness measure is confounded. The pass was budget-limited.
Assessed Supported
verdict confidence 0.75 · credence 0.70
The claim rests mainly on two experimental programmes. Khan et al. (ICML 2024) ran debates on the QuALITY reading-comprehension task, where the debaters can see a story and the judge cannot. They measured persuasiveness as an aggregate Elo rating derived from cross-play win rates, meaning how often judges picked a debater's answer, and raised it by best-of-N sampling and critique-and-refine. Higher-Elo debaters produced higher judge accuracy in self-play debates, both for LLM judges and for non-expert human judges. The same paper found the opposite under consultancy, where a single advisor argues: judges became less accurate as the consultant grew more persuasive. So the relationship belongs to the adversarial debate format, not to persuasion in general. Kenton et al. (NeurIPS 2024) tested the relationship across nine tasks with weaker LLM judges. On extractive tasks, where the judge cannot see the source text, they found it again, but described the effect as "relatively small". On closed QA and multimodal tasks, where the judge and debaters have the same information, the debaters differed little in Elo or in the judge accuracy they produced, so the trend there is weak or not established. Two cautions apply to how persuasiveness was measured. First, Elo comes from wins in front of the same kind of judge whose accuracy is then scored. Second, the stronger base models are both more persuasive and more capable, so persuasiveness and general capability are confounded. Neither programme tested debaters optimised to deceive, and the evidence comes from a small number of task families. Overall, the evidence favours the claim for LLM debate on tasks where the judge lacks information the debaters have. It does not establish the claim for symmetric-information tasks, and the size of the effect is uncertain.
Claim entered the graph