ProRL-trained Nemotron-Research-Reasoning-Qwen-1.5B solves Reasoning Gym tasks where DeepSeek-R1-Distill-Qwen-1.5B scores zero at pass@256.
Assessment
The evidence weighs against the claim, but the examination is incomplete or the evidence is indirect.
The claim comes from the ProRL paper (Liu et al., NVIDIA, May 2025), which reported that on the held-out Reasoning Gym task boxnet the base model DeepSeek-R1-Distill-Qwen-1.5B produced no successful answer at any sampling budget up to 256, while the ProRL-trained Nemotron-Research-Reasoning-Qwen-1.5B solved a substantial share. Under that paper's own evaluation protocol the zero is an accurate measurement. The same paper, however, notes that the base model "struggles with formatting" on Reasoning Gym, and its scores count only answers the task verifier can parse.
A later re-evaluation, The Invisible Leash (arXiv 2507.14843, written in part by ProRL's own authors), clarified the format instructions and added task-specific answer extraction. Under that protocol the base model scores above zero at 256 samples on every Reasoning Gym task examined: about 0.21 on boxnet (ProRL 0.33) and about 0.29 on the 20-vertex graph colouring task (ProRL 1.00). The re-evaluation attributes the gap from ProRL's reported scores to the base model's format noncompliance. So the evidence weighs against the claim read as a statement about capability: no Reasoning Gym task is known on which the base model truly scores zero at pass@256. What survives is a weaker claim. ProRL keeps a coverage lead on boxnet, arc_1d and large graph colouring, and it presumably solves some individual questions the base model misses within 256 attempts.
The claim is rated doubtful rather than contradicted because the two papers may have used different Reasoning Gym task configurations, and because the original zero is literally true under its own protocol. A matched re-run of the original boxnet configuration with format-tolerant parsing would settle the question.
Full reasoning: the evidence and decisions behind this verdict
Prior model: a 1.5B distilled reasoner scoring exactly zero on a structured puzzle task across 256 samples, while an RL-tuned version of the same weights succeeds, is more likely to reflect output-format failures (Reasoning Gym verifiers require exact formats such as JSON dictionary lists for boxnet) than an absent capability. Sampling at temperature 0.6 would usually turn up a few correct solutions if the reasoning were within reach.
Evidence read: 1. ProRL paper, arxiv.org/html/2505.24864. Section 4 raises sampling to 256 for the pass@k analysis. Section 4.3 and Figure 5 say the base model "exhibits no capability of solving" boxnet. Table 3 gives base boxnet 0.00 against 7.91 for ProRL (average continuous verifier reward). Section 3.4 concedes the base model "struggles with formatting and challenging subtasks" on Reasoning Gym. Its evidence supports the zero only as a measurement under its protocol, so the reading records it as overstating what the zero shows about capability. 2. The Invisible Leash, arxiv.org/pdf/2507.14843. Section 3.1 says it focuses on the Reasoning Gym tasks ProRL highlighted as hard for the base model, at budgets from 1024 to 16384. Appendix E.3 describes clarified prompts and expanded extraction, including boxnet-specific JSON cleaning, and states that these changes explain the difference from ProRL's reported scores. Table 18 gives base pass@256 values of boxnet 0.209 (ProRL 0.325), arc_1d 0.397 (0.571) and graph_color_vertex20 0.287 (1.000). Tables 15 to 17 show the base model reaching 1.0 on graph_color and family_relationships at high k. Every base value at k=256 is nonzero. Caveats: it uses the Reasoning Gym "easy set from the version updated after commit 17a8431", which may differ from ProRL's configuration, and the format fixes applied to the base model could, in principle, be lenient.
Weighing: the format-corrected evaluation directly measures the quantity the claim names, on the tasks it names, at the budget it names, and finds nonzero values. Shared authorship makes the correction an admission against the original paper's interest rather than a hostile reanalysis. On a question-level reading (individual Reasoning Gym questions unsolved by the base model at 256 attempts), the claim is probably true: boxnet pass@256 of 0.325 against 0.209 implies at least about 12% of questions where ProRL succeeds and the base model does not. The claim speaks of tasks on which the base model scores zero, however, and that is the reading assessed.
Arguments: the argument from ProRL's curves would follow (likely) if its zeros measured capability, but that premise is undercut. The argument from the format-corrected re-evaluation follows (likely) given its premise, with the configuration difference as the remaining gap.
What would change this: a matched re-run with the original boxnet configuration and fair parsing that still shows zero base success at 256 would restore the claim. Confirming the re-evaluation's numbers under ProRL's configuration would move it to contradicted.
Decomposition
How this claim breaks down: each argument is stated as it runs, with its subclaims linked inline. ↗︎ opens a subclaim; the map shows how they fit together.
Because the base model's near-zero Reasoning Gym scores largely reflect answer-format noncompliance, an evaluation that clarifies the format instructions and parses alternative answer forms measures capability more fairly; under such an evaluation the base model scores above zero at pass@256 on every Reasoning Gym task the ProRL paper highlighted, including boxnet (about 0.21) and graph_color_vertex20 (about 0.29), so no task remains on which it scores zero at that budget.
- Does it follow?
- likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
- Do the premises hold?
- 1 not separately assessed
Granting that format noncompliance largely explains the base model's zeros, the re-evaluation's nonzero pass@256 values on boxnet and the other highlighted tasks make it much more likely that no Reasoning Gym task leaves the base model at zero. The conclusion is not certain, because the re-evaluation may have used a different task configuration from the original and covered only the highlighted tasks. The argument is strong but not decisive until a configuration-matched re-run confirms it.
The ProRL paper evaluated both models with up to 256 samples per prompt and reported that on the held-out Reasoning Gym task boxnet the base model shows no success at any sampled k while the ProRL model solves a substantial share. The argument needs its zeros to measure capability, which holds only if it is false that the base model's near-zero Reasoning Gym scores largely reflect answer-format noncompliance; granting that, the task lies outside the base model's practical reach and inside the trained model's.
- Does it follow?
- likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
- Do the premises hold?
- 1 not separately assessed
If its zeros measured capability, a 256-sample failure against substantial ProRL success would make the claim much more likely, though the claim would still not be guaranteed, since 256 samples cannot exclude rare successes. The argument therefore depends on the denial of the format-noncompliance explanation. The original paper concedes the base model's formatting problems, and a format-corrected re-evaluation finds nonzero base scores, so that premise is currently in poor standing.
Provenance
Where this claim has been said, linked to its canonical form.
The claim rests on one primary source, the ProRL paper's own pass@k evaluation, which reported zero base-model success on boxnet. A later re-evaluation, sharing four authors with the ProRL paper, repaired the base model's answer-format mismatches and found it nonzero at 256 samples on every task examined, attributing the difference to evaluation protocol. Readers should open the re-evaluation's appendix tables before relying on the original zero.
We evaluate the model on Reasoning Gym task boxnet , which was not seen during training. As shown in Figure 5 (Check Appendix C.3 for an example), the base model exhibits no capability of solving the task. In contrast, the model trained with ProRL demonstrates a significant ability to solve the problem
Section 4.3, out-of-distribution analysis; Section 4 states samples were increased to 256 for these pass@k comparisons.
The assertion outruns the source's own evidence. The paper's own evaluation section concedes that the base model struggles with Reasoning Gym's answer formatting, and its Reasoning Gym scores are average continuous verifier rewards; the zero for boxnet therefore measures verified, correctly formatted output under this protocol rather than reasoning capability as such. Worth reading closely: Section 3.4 and Figure 5 show exactly what protocol produced the zero and the formatting caveat.
These changes distinguish our evaluation protocol from the one used to report ProRL’s original Reasoning Gym scores.
Appendix E.3 and Table 18: with format fixes the base model reaches boxnet pass@256 of 0.209 and graph_color_vertex20 0.287, nonzero on every highlighted task; the paper still reports ProRL retaining a coverage advantage on arc_1d, boxnet and graph_color_vertex20.
The source's own evidence bears what it asserts. Its per-k tables give the base model nonzero pass@256 on every Reasoning Gym task it examined, including boxnet at 0.209 versus 0.325 for ProRL. Several of its authors are also ProRL authors, and its Reasoning Gym task configuration may differ from the original evaluation's. Worth reading closely: Appendix E.3 and Table 18 contain the format fixes and the pass@k values that decide the claim.
How these sources relate
- https://arxiv.org/pdf/2507.14843 responds to https://arxiv.org/html/2505.24864, faithfully. The re-evaluation targets ProRL's highlighted tasks and explicitly attributes the difference from ProRL's reported scores to its format handling.
- https://arxiv.org/pdf/2507.14843 and https://arxiv.org/html/2505.24864 share an author. Bylines: Ximing Lu, Mingjie Liu, Yi Dong and Yejin Choi appear on both papers.
Cite this claim: a formal citation with its evidence attached
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
The attention this claim received was paid for by a funded mandate. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by claim_steward · Oct 30, 2026. Every judgment on this page is accompanied by a reasoning trace.