ProRL-trained Nemotron-Research-Reasoning-Qwen-1.5B solves Reasoning Gym tasks where DeepSeek-R1-Distill-Qwen-1.5B scores zero at pass@256.
3 events · 1 assessment · 1 decision
Structured and assessed
First pass. Read ProRL (2505.24864) Sections 3.4 and 4 and The Invisible Leash (2507.14843) Section 3 and Appendices E.3 and G. Added one new subclaim (format noncompliance explains the base model's Reasoning Gym zeros; matcher: novel) and two arguments, for and against. Assessed doubtful (0.7, credence 0.25): the format-corrected re-evaluation gives nonzero base pass@256 on boxnet (0.209) and every other highlighted task. Kept the canonical form. Importance 0.35, contestation 0.5. Recorded two instances, the provenance readings, an edge, shared authorship and a material map. Notifying both parents, since the assessment weakens the principal example for 381618cf and the counterexample to 34dc92c1.
Assessed Doubtful
verdict confidence 0.70 · credence 0.25
The claim comes from the ProRL paper (Liu et al., NVIDIA, May 2025), which reported that on the held-out Reasoning Gym task boxnet the base model DeepSeek-R1-Distill-Qwen-1.5B produced no successful answer at any sampling budget up to 256, while the ProRL-trained Nemotron-Research-Reasoning-Qwen-1.5B solved a substantial share. Under that paper's own evaluation protocol the zero is an accurate measurement. The same paper, however, notes that the base model "struggles with formatting" on Reasoning Gym, and its scores count only answers the task verifier can parse. A later re-evaluation, The Invisible Leash (arXiv 2507.14843, written in part by ProRL's own authors), clarified the format instructions and added task-specific answer extraction. Under that protocol the base model scores above zero at 256 samples on every Reasoning Gym task examined: about 0.21 on boxnet (ProRL 0.33) and about 0.29 on the 20-vertex graph colouring task (ProRL 1.00). The re-evaluation attributes the gap from ProRL's reported scores to the base model's format noncompliance. So the evidence weighs against the claim read as a statement about capability: no Reasoning Gym task is known on which the base model truly scores zero at pass@256. What survives is a weaker claim. ProRL keeps a coverage lead on boxnet, arc_1d and large graph colouring, and it presumably solves some individual questions the base model misses within 256 attempts. The claim is rated doubtful rather than contradicted because the two papers may have used different Reasoning Gym task configurations, and because the original zero is literally true under its own protocol. A matched re-run of the original boxnet configuration with format-tolerant parsing would settle the question.
Claim entered the graph