At high k, base models match or exceed RLVR-trained models in pass@k on math, code, and visual reasoning benchmarks.
5 events · 2 assessments · 2 decisions
Reassessed
Subclaim (ProRL zero-base counterexample) assessed doubtful. Change absorbed: counter-argument already slight; now weaker. Status stays supported, confidence 0.75, credence 0.78 -> 0.80. Argument re-evaluated (still slight, premise now doubtful). Importance 0.45, contestation 0.6. No dependents notified: change is not material to status.
Reassessed: still Supported
verdict confidence 0.75 · credence 0.80
Structured and assessed
First pass. Recorded the Yue et al. abstract as an affirming instance. Attached the ProRL Reasoning Gym claim as a contradicting subclaim under an against argument, rated slight. Linked the claim on whether large-k pass@k is a valid measure as related. Decided not to mint the CoT-pass@k claim (the Matcher found it novel) because it bears on interpretation rather than on the plain-pass@k fact; it is covered in the prose. Assessed supported at 0.75 confidence, credence 0.78. Importance 0.45, contestation 0.6. No notification sent, since this is the first assessment and the parents were already built on this premise.
Assessed Supported
verdict confidence 0.75 · credence 0.78
The claim rests chiefly on Yue et al., "Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?" (NeurIPS 2025 oral), which compared base models with their RLVR-trained versions across several model families, six RL algorithms, and math, coding, and visual reasoning benchmarks. RLVR models win at small k, but the pass@k curves cross and the base models score higher at large k (typically in the hundreds of samples). The crossover is consistent with a simple mechanism: RLVR sharpens the output distribution toward already-reachable correct solutions, raising pass@1 while losing some coverage. For standard-length RLVR on the benchmarks studied, the finding is well supported. Two qualifications limit how far the claim generalizes. First, it is not universal across RL regimes: prolonged RL work such as ProRL reports a ProRL-trained model solving Reasoning Gym tasks on which its starting model scores zero at pass@256, although that evidence comes from synthetic tasks and a distilled starting model, not the standard benchmarks. Second, high-k pass@k on answer-checked math credits lucky guesses with flawed reasoning. Wen et al. (arXiv 2506.14245) report that under a metric requiring a correct chain of thought, the RLVR model stays ahead at large k. That bears on what the crossover means, but not on whether it happens under plain pass@k, which is what the claim states.
Claim entered the graph