https://arxiv.org/pdf/2605.27483

Debate Helps Weak Judges Reward Stronger Models Ethan Elasky∗ Palaestra Research elasky2024@berkeley.edu Frank Nakasako∗ Palaestra Research frank@palaestraresearch.org Naman Goyal Independent Abstract Despite theoretical promise, debate as a scalable oversight protocol has produced mixed empirical results: gains in some settings, and null effects in others, especially when the judge does not have information hidden from it. We study proposer-critic debate in a stronger-debater/weaker-judge setting on programmatically verifiable code and logic tasks. Debate helps the judge over a consultancy baseline when the critic provides a usable advantage: the critic’s classification ability must exceed the judge’s, and the judge must treat critic speeches as claims to verify rather than testimony to summarize. On the three of five pairings where the condition holds, proposer-critic debate’s gains are statistically significant over consultancy, and these pairings are the most capable model pairings. On the two non-responder pairings in our set, debate produces null effects, and judge verification rates drop by tens of percentage points once a critic enters the transcript. In these cases the critic’s binary-classification ability and the judge’s are within noise of each other, and the critic’s disagreement is parsed as testimony rather than a claim to check. Ablating rebuttal rounds from debate produces no measurable change in judge performance: a single independent critique recovers the bulk of debate’s benefit at lower inference cost. These findings suggest a cheaper primitive for training- free scalable oversight in verifiable domains (answer, critique, judge) and a pre- deployment audit (does the critic beat the judge, and will the judge verify it?) that predicts when debate will help. 1 Introduction State-of-the-art training regimes enable LLM performance rivalling the best humans on verifiable domains [OpenAI, 2024, DeepSeek-AI et al., 2025]. Frontier and open-weight models now succeed at complicated programming tasks [Jimenez et al., 2024], broad knowledge tasks [Wang et al., 2024], and graduate-level science reasoning [Rein et al., 2023], and only the hardest single-turn programmatically verifiable benchmarks remain unsaturated [Phan et al., 2025, Glazer et al., 2024, Chollet et al., 2025]. However, fuzzier domains, whose outcomes are harder to verify programmatically, such as creative writing and scientific experiment design, lag behind [Xu et al., 2025, Rashkin et al., 2025, Si et al., 2025]. Post-training in these areas relies substantially on human expert labels and/or LLM ratings for reward assignment [Moonshot AI, 2025, Yang et al., 2025], typically via reinforcement learning from human feedback (RLHF; Christiano et al. 2017) or AI feedback (RLAIF; Bai et al. 2022). Both approaches have persistent problems. Human labels remain expensive and are typically collected ∗Joint first authors. Preprint. arXiv:2605.27483v1 [cs.CL] 26 May 2026

offline [Ouyang et al., 2022, Casper et al., 2023], which delays training and can make supervision increasingly off-policy. Machine labels suffer from quality issues and adversarial brittleness, failing to provide a fully stable training signal [Casper et al., 2023, Wen et al., 2024]; and methods that mitigate these issues often reintroduce human experts through rubric design and data curation [Moonshot AI, 2025]. Prior work proposes debate as a potential alternative, with theoretical results suggesting that debate causes strong arguments to surface [Irving et al., 2018, Brown-Cohen et al., 2023]. Compared with RLAIF-style baselines in which a model judge evaluates an answer directly or after one-sided self- defense, debate adds an independent critic who can surface falsifying evidence. Empirical findings have been uneven: Khan et al. [2024] found debate gains in hidden-evidence settings, while Kenton et al. [2024] reported weak or null effects in visible-information weak-judge settings closer to the one we study. We do not attempt to fully reconcile these results—protocol, task, and judge differences across that literature make a direct comparison difficult, and our diagnostic does not cleanly translate to two-debater assigned-position protocols. We instead identify a measurable condition under which proposer–critic debate reliably helps in our setting, and we show, through transcript-level behavior reviews, why it does. On code and logic—calibrated verifiable testbeds chosen so that we can audit reward signal against ground truth—proposer–critic debate raises macro-F1 over consultancy on three of five model pairings and shows no effect on the other two. The split is governed by a single condition: debate helps only when the critic provides a usable advantage. That condition has two sub-parts, both of which must hold. First, the critic’s classifier macro-F1 must exceed the opening-only consultancy judge’s—there has to be a margin worth importing; raw judge capability and the critic’s generator–verifier gap do not predict the split, but this critic–judge classifier gap does. Second, the judge must treat the critic’s stance as a claim to verify rather than testimony to summarize. Behavior reviews of the transcripts (§4.2, Appendix L) show the mechanism: on responder pairings both sub-parts hold and judges import the critic’s signal; on the two non-responder pairings in our set, both sub-parts fail together—the critic’s and judge’s binary-classification ability are within noise of each other, and verification rates drop by tens of percentage points once a critic enters the transcript, with the critic’s disagreement parsed as testimony rather than a claim to check. We flag a small-N caveat: with five pairings, the gap diagnostic is a hypothesis well-supported within our experimental set, not a law established at scale. We also note that with ARC-AGI-2 substantially harder than Code Contests, our pairings do not let us cleanly separate the critic–judge gap from broader capability differences; the two conditions we identify (judge verification behavior and critic–judge classifier gap) are themselves capability properties, just more specific ones than raw solve rate. Whether the same mechanism transfers to fuzzier domains where programmatic verification is unavailable is a hypothesis warranting further work, not a finding from our experiments. Our main contributions are as follows: • In a visible-information weak-judge setting, proposer–critic debate lifts macro-F1 by 7–16pp on three of five CodeContests+ and ARC-AGI-2 pairings, and gives no benefit on the other two. • A single condition governs when debate helps—the critic provides a usable advantage— with two sub-parts that must jointly hold: (i) the critic’s classifier macro-F1 exceeds the opening-only consultancy judge’s (the more obvious generator–verifier gap does not predict the split), and (ii) the judge treats the critic’s stance as a claim to verify rather than testimony to summarize. On the two non-responder pairings, both sub-parts fail together. • At test time, almost all of the lift comes from the critic’s opening stance; rebuttals neither help nor hurt. Their training-time value is out of scope here. 2 Background 2.1 Scalable Oversight A central challenge in scalable oversight is getting reliable reward signal when human supervision is scarce, limited, or less capable than the systems being supervised. Current post-training pipelines often use reinforcement learning from human feedback (RLHF) or AI feedback (RLAIF), but both can be unstable: human labels are expensive and often offline, while machine labels can be brittle 2

or exploitable. Debate [Irving et al., 2018] offers a different reward-labeling protocol. Instead of asking a judge to evaluate an answer in isolation, debate elicits adversarial information that may help a weaker judge decide whether a stronger model’s answer is correct. This framing connects to the broader scalable oversight literature. Cotra [2021]’s sandwiching paradigm and Bowman et al. [2022]’s question-answering experiments study whether weaker or non- expert supervisors can be aided by models to approximate expert judgment. Recent work extends this question to richer oversight protocols, including market making [Hubinger, 2020], and self-critique [Saunders et al., 2022]. Model-generated critiques have been shown to help human evaluators find flaws in summaries and model-written code, though critics can hallucinate problems and therefore remain imperfect supervision aids [Saunders et al., 2022, McAleese et al., 2024]. Prior work also finds that larger models produce outputs that are harder to critique while becoming better critics themselves [Saunders et al., 2022], motivating scale as a possible source of critic–judge gaps. More recently, Engels et al. [2025] establish scaling laws for scalable oversight across four games (mafia, debate, backdoor code, wargames), finding that debate has the highest nested-oversight success rate at a fixed Elo gap among the protocols tested, though all protocols’ rates decline as the overseen system grows stronger. We study a narrower and more training-relevant question: does debate give a weaker judge a better reward label than consultancy or answer-only evaluation when the answer comes from a stronger model? Our setting differs from prior scalable-oversight experiments in three ways. First, we evaluate debate as reward labeling, not mainly as human assistance or answer recovery. Second, the judge sees the problem and proposed answer; in debate it also sees the critic’s stance or argument. The challenge is not to recover hidden source material, but to verify or falsify a visible answer. Third, we use an open proposer–critic format. The proposer generates its own solution, and the critic may agree or disagree. This makes the critic closer to a verifier than an advocate, and makes the consultancy comparison cleaner: consultancy lacks a critic, but is not weakened by forced false advocacy [Roger, 2024, Kenton et al., 2024]. 2.2 Debate We focus on judge-mediated debate as a scalable oversight protocol, distinct from multi-agent “debate” methods that use round-robin discussion and majority voting to improve factuality or reasoning [Du et al., 2023, Liang et al., 2023]. In judge-mediated debate, participants present arguments to a judge whose task is to decide which answer or claim is correct. Irving et al. [2018] provide an early formulation of debate for scalable oversight, supported by image-classification experiments in which debaters reveal information that helps a limited judge classify images more accurately. Empirical debate work often studies cases where the judge lacks decisive evidence. In hidden-passage QuALITY experiments, debate can beat consultancy by surfacing passage evidence for the judge [Pang et al., 2021, Michael et al., 2023, Khan et al., 2024]. That advantage weakens when the passage is visible, which suggests that some gains come from evidence retrieval rather than better verification [Khan et al., 2024, Parrish et al., 2022a,b]. Kenton et al. [2024] extend this line to weak-judge/strong-agent settings in mathematics, coding, logic, and multimodal reasoning, and find smaller debate gains over consultancy. This distinction is central here: in code and ARC-AGI-style tasks, the judge sees the task and candidate answer. The question is whether a critic helps the judge check that answer. A second difference is stance assignment. Some debate and consultancy comparisons force agents to defend externally specified positions. This can make consultancy artificially weak when the consultant must defend an answer it would not endorse. Kenton et al. [2024]’s open-debate setting lets one debater choose its position while fixing the other in opposition. We go further: the proposer generates an answer, and the critic decides whether to agree or disagree. Open consultancy is therefore a cleaner baseline: it lacks a critic, but is not penalized by forced false advocacy [Roger, 2024]. Theoretical work studies when debate can provide robust oversight guarantees. Brown-Cohen et al. [2023] introduce doubly-efficient debate, designing protocols in which the honest debater requires only polynomial computation while the dishonest debater gains no advantage from exponential resources. Brown-Cohen et al. [2025] further address the obfuscated-arguments problem through prover-estimator debate, which aims to reduce the risk of arguments that are persuasive but difficult for the judge to evaluate. We distinguish our empirical protocol from these theoretical variants, 3

−10 0 10 20 30 40 Class-specific F1 lift (debate over consultancy, pp) Gemini 3.1 Pro / Flash Opus 4.6 / 4.5 Qwen3.5 122B / 35B Qwen3.5 35B / Q3 4B gpt-oss 120B / 20B Correct proposer Incorrect proposer +7.2 pp [3.2, 12.3] +20.8 pp [9.1, 34.1] +6.8 pp [2.5, 11.1] +24.7 pp [8.3, 40.7] +4.4 pp [2.9, 5.8] +23.7 pp [18.0, 29.6] +0.9 pp [-1.2, 3.1] -1.3 pp [-5.9, 3.2] +1.0 pp [-0.2, 2.2] +1.8 pp [-1.6, 5.0]Figure 3: Class-specific F1 reveals where debate helps. All three responder pairings show large incorrect-class lifts; at the verdict-share level the dominant stratum is incorrect-proposer rejection on Qwen3.5-122B/35B and Gemini and correct-proposer acceptance on Opus, reflecting Opus 4.5’s opposite no-transcript prior (per-pairing stratum-level agreement shifts in Appendix D). Two patterns drive the result. First, when debate helps, it mostly cuts false positives: judges stop endorsing wrong proposer answers. Second, debate helps only when the critic’s classifier signal beats the judge’s answer-only baseline and the judge checks that signal rather than merely summarizing it. Debate’s gains concentrate on rejecting wrong proposers. On the three responder pairings, moving from consultancy to debate cuts the judge’s false positive rate (endorsing an incorrect proposer answer) by 7.3–10.0pp, while the false negative rate stays roughly flat (Appendix L, Table 16). This asymmetry matters for reward modeling: false positives teach a policy that flawed reasoning is acceptable, while false negatives only withhold reward from good answers. Consultancy raises agreement with the proposer indiscriminately. The critic gives the judge concrete grounds to disagree—a held-out-grid inconsistency on ARC, a counterexample input on code—without inducing fabricated objections to correct answers. Class-conditional F1 shows the same pattern (Figure 3). Incorrect-class F1 rises by +20.8, +24.7, and +23.7pp on Gemini, Opus, and Qwen, while correct-class F1 rises only +4–7pp. We use class-F1 because it is symmetric in the two error types and matches the metric used for protocol evaluation. Opus 4.5 has the opposite no-transcript prior (its baseline tendency on the proposer’s answer alone, with no advocacy) from the other two responder judges—it under-endorses correct proposers, where Qwen and Gemini over-endorse—so its largest raw verdict-count movement is accepting correct answers rather than rejecting wrong ones (Appendix D, Table 5). The class-F1 view and verdict-count view are consistent: both error types fall, but the dominant raw-count stratum depends on the judge’s prior. The ARC behavior reviews make the mechanism concrete. One-sided proposer explanations help the judge recognize correct transformations, which cuts false negatives but also increases over- endorsement. The critic’s opening adds the missing skepticism: it gives the judge an alternative rule or held-out-grid inconsistency to check. Appendices J.1 and J.2 walk through one responder transcript on each domain, with the complete debates reproduced in Appendices M.1 and M.2. Prompt robustness. The Qwen3.5-122B/35B Code Contests lift is partly prompt-sensitive. Under a simpler judge prompt the consultancy baseline rises and the debate–consultancy gap shrinks to +7.3pp (still p < 0.001); opening-only and full debate are essentially unchanged. We treat +7.3pp as the prompt-robust estimate on this pairing. The split itself is robust on the two pairings we reran with the simpler prompt: Qwen3.5-122B/35B remains a responder (full-debate lift +7.3pp, p < 0.001) and gpt-oss-120B/20B remains a non-responder (full-debate lift −1.6pp, p = 0.27); we did not rerun the ARC pairings or weak Qwen under the simpler prompt (Appendix G). Compute control. The lift is also not a compute artifact, at least on Code Contests: doubling consultancy compute by running two independent consultants instead of one does not close the gap to debate (Qwen3.5-122B/35B: ∆ = +0.021, p = 0.24; Qwen3.5-35B/Qwen3-4B: ∆ = −0.056, p = 0.003). The added second consultant supplies more advocacy but no independent verification signal. ARC compute-matched baselines are deferred to future work (Appendix E). 7

−5 0 5 10 15 20 25 30 Critic − Proposer accuracy (pp) 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 Macro-F1 lift (pp) A Gemini Opus Qwen 122B/35B Qwen 35B/4B gpt-oss −5 0 5 10 15 20 Critic − NT-judge macro-F1 (pp) B Gemini Opus Qwen 122B/35B Qwen 35B/4B gpt-ossFigure 4: Debate’s lift over consultancy is largest where the critic’s classifier macro-F1 exceeds the lone judge’s. Panel A shows the generator-verifier gap—the same model’s accuracy as a verifier minus its accuracy as a Direct QA generator—which is positive on every pairing. Panel B shows the critic’s classifier macro-F1 minus the opening-only consultancy judge’s, a non-debate analog of RLHF-style verifier evaluation. Debate gains track Panel B, not Panel A: the gen-verify gap is necessary but not sufficient. 4.2 Debate helps when the critic provides a usable advantage A single condition separates responders from non-responders: the critic must provide a usable advantage. That condition has two sub-parts, both of which must hold. First, the critic’s classifier macro-F1 must exceed the opening-only consultancy judge’s—a margin the judge could in principle import (Figure 4, Panel B). Second, the judge must actually behave as a verifier rather than a summarizer of testimony. Both sub-parts hold on responder pairings; both fail together on the two non-responder pairings in our set. On every responder pairing, judges that see the critic—in opening-only debate and full debate—match the critic’s macro-F1 within ±3pp. In other words, they import the critic’s signal. Critic errors are themselves mostly over-endorsements rather than fabricated objections, especially on ARC (Appendix L, Table 17), so importing the critic mostly helps the judge reject wrong answers. A more obvious diagnostic fails. On every pairing, the same model is better at verifying than generating: solo solve rates of 60–74% become 76–91% as a verifier (Figure 4, Panel A). But this generator–verifier gap is not enough. The two non-responder pairings also have large gaps, but their critics offer no useful margin over the answer-only judge—the critic–judge classifier gap is slightly negative on gpt-oss and essentially zero (+0.7pp) on weak Qwen—so the judge has no better signal to import. The judge’s behavior shows the same split. Responders treat the critic’s opening as something to check. Non-responders treat it as testimony to summarize. Table 2 shows this directly. The three responder pairings verify at high rates across formats (78–100%; only Gemini opening-only consultancy is below 88%). Both non-responder pairings verify less once a critic is in the transcript: gpt-oss falls from 67–82% across the two consultancy formats to 16–31% across the two debate formats, and weak Qwen falls from 61–67% to 25–53%. Their null results are failures to use the signal, not failures to receive it. Appendix J.3 walks through one such non-responder transcript on gpt-oss; the full debate is reproduced in Appendix M.3. Behavior-review methodology. Verification rates score each judge response against a role- and class-specific questionnaire (does the verdict run its own derivation, does it cite participant statements, does it construct an explanation for an incorrect endorsement). The ARC review covers the full population on the two reviewed pairings; the Code Contests review is a paired, stratified sample of ∼20 transcripts per TP/FP/TN/FN stratum anchored to the Qwen3.5-122B/35B responder run, with the same task IDs scored for the gpt-oss and weak-Qwen comparisons. Reported rates pool the TP and TN classes weighted by their reviewed counts; the full coding scheme is in Appendix L. 4.3 Rebuttals add little at test time Most of debate’s gain comes from the critic’s opening, not from later rebuttals. On every responder pairing, full debate and opening-only debate are statistically indistinguishable on macro-F1 and agree 8

redefined DATA inside the function, arguing it was “safer”. The other followed the spec. Pytest rejected both. The fix is the minimal one: define DATA in the test file. No implementation needs to change. Example 3: BigCodeBench/664, spec-test scope mismatch. The spec asks the function to plot monthly sales for multiple products “highlighting variability with standard deviation shading”. It does not say whether the shading is per-product (each line gets its own plus-minus sigma band) or cross-product (one band over the product mean). The tests silently assume per-product. Separately, test_monthly_ticks filters x-ticks with isinstance(tick, (int, np.integer)), but mat- plotlib emits float ticks by default, so the test rejects every valid implementation on macOS matplotlib 3.8 and above. The debate transcripts split. Debater A produced a single-band plot that the judge endorsed but that pytest failed. Debater B produced per-product bands that pytest accepted but with float ticks that test 4 still rejected. The fix specifies per-product shading and rewrites the tick check to round and range-filter, not type-filter. Example 4: BigCodeBench/939, filesystem-impossible assertion. Given three input files whose sanitized names all collapse to ’txt’, the original test_only_special_characters asserts the output is [’txt’, ’txt’, ’txt’]. That is a state no real filesystem can ever produce without at least one write collision. Every correct implementation produces unique names by suffixing, which the test rejects. Both debaters produced uniqueness-enforcing code and both were correctly endorsed by the judge and incorrectly failed by pytest. The fix replaces the equality assertion with property checks: three files returned, all names unique, all alphanumeric, all containing the base txt. This preserves difficulty while removing the physical impossibility. H.5 Why These Fixes Matter for Debate Under direct QA or consultancy, a broken test hurts both the correct and the incorrect participant symmetrically. The judge sees only an answer, and in consultancy a single explanation, and accepts or rejects based on its own prior. Under debate, the critic is rewarded for surfacing the broken test. It can truthfully argue that the proposer’s code will fail pytest because it will, but for the wrong reason. The adversarial format amplifies ground-truth noise rather than filtering it out. Curating the 552 spec-ambiguous tasks in particular removes critics’ free wins. Every remaining spec admits a single reading, so a critic that argues against a reasonable proposer must point at a real code defect. This is also why spec_too_easy, which covers only 6 tasks, was deprioritized during curation. Our goal is not to make the benchmark harder. It is to make spec-conformance and test-conformance coincide. BCB+ v1.0.0 is not a smaller or harder BCB. It is a cleaner BCB on which debate’s reward signal measures what it is supposed to measure. H.6 Reproducibility and Release • Dataset: a public HuggingFace repository released with the camera-ready, tagged v1.0.0. Per-task version history is preserved so any task can be reverted to its BCB original for ablation. • Static evaluator: a model-heterogeneous evaluator stack with fallback parsing for mal- formed review outputs. • Task-level reviewer: assisted review for examples not resolved by the static pass, with human escalation for ambiguous cases. • Test validator: pytest on a pinned Python 3.10 environment isolated per task. Reference- solution runs provide ground truth for every proposed fix before it is pushed. • Infrastructure: the released artifact will include the evaluation, triage, verification, and application scripts needed to reproduce the curation state. Each fix is tagged with the run and transcript category that triggered it. I Why we scope to code and ARC: misaligned final-answer transcripts The main experiments restrict to CodeContests+ and ARC-AGI-2, where the oracle—hidden tests or exact-grid match—is faithful to task intent. Final-answer-graded math is a natural-looking next 23