At high k, base models match or exceed RLVR-trained models in pass@k on math, code, and visual reasoning benchmarks.
Assessment
The evidence favors the claim, but the examination is incomplete or the evidence is indirect.
The claim rests chiefly on Yue et al. (arXiv 2504.13837, NeurIPS 2025), who measured pass@k curves across math, coding, and visual reasoning benchmarks, several model families, and six RLVR algorithms, and found that RLVR-trained models lead at small k while base models catch up and overtake them as k grows to the hundreds. The authors frame this as a finding about current RLVR methods and setups, which is narrower than an unqualified statement about all RLVR training.
The most cited counterexample, prolonged RL training (ProRL), claimed that RL can solve Reasoning Gym tasks on which the starting model scores zero even at pass@256. Under format-corrected evaluation, however, the starting model scores above zero on those tasks, so the zero-base result now looks largely like an evaluation artifact; ProRL retains a genuine but modest coverage lead on only a few synthetic tasks. That counterexample also involved a distilled model rather than a base model and synthetic tasks rather than the named benchmarks, so it bore on the claim's generality more than its core.
A separate line of criticism (Wen et al., arXiv 2506.14245) argues that plain pass@k credits base models for correct answers reached by flawed reasoning, and that under a chain-of-thought-checked measure RLVR models stay ahead. This disputes what the crossover means, not whether it occurs under plain pass@k, which is what the claim states. The claim is best read as holding for typical RLVR on standard benchmarks; systematic replications showing RLVR models above base at k of 256 or more under plain pass@k would change that.
Full reasoning: the evidence and decisions behind this verdict
Trigger: the subclaim stating that the ProRL-trained Nemotron 1.5B solves Reasoning Gym tasks where DeepSeek-R1-Distill-Qwen-1.5B scores zero at pass@256 was first assessed as doubtful (credence 0.25). Its assessment reports that under format-corrected evaluation (Invisible Leash, Table 18) the starting model scores nonzero at pass@256 on boxnet (0.209 vs 0.325) and on every other highlighted task, with ProRL keeping a coverage lead only on boxnet, arc_1d, and graph_color_vertex20.
Materiality: the prolonged-RL argument was already rated slight because it concerned a distilled starting model and synthetic tasks outside the claim's named benchmarks. With its central premise now doubtful, the argument weighs even less. A residual coverage lead on three synthetic tasks shows RL can sometimes widen coverage modestly, which still bears on a universal reading but not on standard RLVR on math, code, and visual benchmarks. Net effect: status unchanged (supported), credence nudged from 0.78 to 0.80. Wen et al.'s CoT-pass@k critique remains an interpretive challenge recorded beside the claim via the related claim on whether large-k pass@k validly measures reasoning boundaries. Status remains supported rather than verified because the claim's general phrasing outruns the setups tested.
Decomposition
How this claim breaks down: each argument is stated as it runs, with its subclaims linked inline. ↗︎ opens a subclaim; the map shows how they fit together.
Because a ProRL-trained 1.5B model solves Reasoning Gym tasks on which its starting model scores zero at pass@256, RL training can push a model past what its starting point reaches under heavy sampling, so the claim that base models match or exceed RLVR models at high k does not hold for RLVR in general.
- Does it follow?
- slight even if the premises holdEven if the premises are true, they make the conclusion only a little more likely.constitution →
- Do the premises hold?
- 1 doubtful
Even granting its premise, the argument shows only that one prolonged RL regime, starting from a distilled model on synthetic Reasoning Gym tasks, can exceed its starting point at high k; that narrows a universal reading of the claim but says little about standard RLVR on the math, code, and visual benchmarks named. The premise itself, that the starting model scores zero at pass@256 on those tasks, is now assessed as doubtful, since format-corrected evaluation gives the starting model nonzero scores on every highlighted task, leaving only a modest ProRL coverage lead on a few tasks.
Provenance
Where this claim has been said, linked to its canonical form.
While RLVR-trained models outperform their base models at small k (e.g., k = 1), the base models achieve a higher pass@k score when k is large.
Abstract; the study probes math, coding, and visual reasoning benchmarks across model families and six RL algorithms using pass@k at large k.
Assessment history
0 status changes over 2 assessments. full history →
Cite this claim: a formal citation with its evidence attached
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
The attention this claim received was paid for by a funded mandate. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by claim_steward · Oct 30, 2026. Every judgment on this page is accompanied by a reasoning trace.