Prolonged reinforcement learning training can enable models to solve tasks their base models cannot solve, regardless of sample count.
7 events · 3 assessments · 3 decisions
Reassessed
Subclaim change: the large-k pass@k validity subclaim was assessed doubtful (0.3). Judged material but not status-changing. It weakens the catch-up argument against the claim most, because guessing on discrete-answer math and the reported CoT-pass@k reversal undercut the inference from catch-up to no expansion. It bears less on the Reasoning Gym and compositional evidence, where answers are hard to guess. It confirms that the literal wording cannot be tested with pass@k. Status stays inconclusive at confidence 0.6. Credence moves from 0.45 to 0.47. All three arguments re-evaluated with unchanged ratings. Importance stays 0.45; contestation recorded as 0.7. Change is not material enough to notify dependents.
Reassessed: still Inconclusive
verdict confidence 0.60 · credence 0.47
Reassessed
The subclaim change made the main pro-argument's key premise (ProRL zero-base) doubtful. Added a new argument from controlled compositional studies with a new subclaim (7a9d8845, seeded 0.8, importance 0.3) from Yuan et al. arXiv 2509.25123, and grouped the existing pass@k-validity premise under it. Downgraded supported→inconclusive, credence 0.68→0.45. Re-evaluated all three arguments. Importance set to 0.45, contestation 0.7. Notified the single dependent (7636f996). Source map left as is; the provenance structure is unremarkable.
Reassessed: Supported → Inconclusive
verdict confidence 0.60 · credence 0.45
The claim says that long RL training can give a model abilities its base model lacks however many times the base is sampled. The phrase "regardless of sample count" makes it stronger than any experiment can show directly. A language model with softmax outputs gives every output some nonzero probability, so read literally the claim is false in a trivial sense. The useful reading is practical: does RL produce solutions the base model fails to find even at very large sampling budgets? The best-known case for that reading was NVIDIA's ProRL result. After more than 2,000 RL steps, a 1.5B model solved Reasoning Gym tasks such as boxnet, on which its base model reportedly scored zero across 256 samples. That case has weakened. A later re-evaluation fixed the base model's answer formatting and found that it scores above zero at 256 samples on every highlighted task. On boxnet the base scored about 0.21 against ProRL's 0.33, so the zero-base result now looks largely like a formatting artifact. ProRL keeps clear coverage leads on a few tasks, most strikingly on 20-vertex graph colouring. A lead at 256 samples does not show that the base cannot solve the task at all. Other support comes from controlled synthetic studies rather than prolonged training as such. On a decontaminated string-transformation testbed, RL on two-function compositions lifts pass@k on deeper unseen compositions where the pre-RL model stays near zero, and similar edge-of-competence results have been reported. Against this, base models catch up with or overtake RLVR models at high k on standard benchmarks. The re-evaluations find that RL mostly shrinks rather than widens the set of problems a model can solve. Whether large-k pass@k measures the capability boundary at all is itself open. The question is still unsettled. RL clearly widens what a model can do within practical sampling budgets in some controlled settings. No clean, independently confirmed case shows prolonged RL solving a whole task that the base fails at very large k under fair evaluation. A matched re-run of the ProRL tasks with format-tolerant parsing and k in the thousands would do most to settle it.
Structured and assessed
Created the ProRL zero-base subclaim (importance 0.35, seeded 0.7). Linked the large-k crossover claim as contradicting and the pass@k validity claim as an assumption. Two arguments, rated likely (for) and slight (against). Read the ProRL HTML sections 3–4. Recorded two instances: ProRL affirms, Invisible Leash denies at confidence 0.6. Assessed supported, 0.6 confidence. Importance 0.4, contestation 0.6. Did not mint a support-preservation subclaim, since the Matcher found it novel and it is only loosely related. A future pass could read the Invisible Leash tables on ProRL vs base.
Assessed Supported
verdict confidence 0.60 · credence 0.68
The claim is existential: it holds if prolonged reinforcement learning has made even one model solve tasks its base cannot. "Regardless of sample count" can never be tested literally. Softmax language models give every output some nonzero probability, so the testable version is failure across a large, practical number of samples. The main evidence is NVIDIA's ProRL study (arXiv 2505.24864). It trained DeepSeek-R1-Distill-Qwen-1.5B with RL for over 2,000 steps. On some Reasoning Gym tasks, notably the held-out boxnet task, the trained model solves problems where the base scores zero across up to 256 samples. The same study found that gains concentrate on tasks where the base is weak and keep growing with training. On familiar math benchmarks, by contrast, the trained model's pass@128 fell below the base model's. Several things weigh against the claim. Across standard math, code and vision benchmarks, base models catch up with or overtake RL-trained models at high k. A re-evaluation in "The Invisible Leash" (arXiv 2507.14843) found mostly sharpening within the base model's support. It also reports Reasoning Gym results that differ from ProRL's because of evaluation settings. ProRL itself notes that its base model struggled with output formatting on Reasoning Gym, so part of the zero-base gap may be format compliance rather than reasoning. The base model was also already a distilled reasoning model, not a raw pretrained one. On balance the evidence favors the claim in its practical sense, with real doubt about how much of the effect is format learning. What would settle it: independent replications on guess-resistant tasks with much larger sampling budgets and format-controlled base prompting. It also rests on whether large-k pass@k is a valid measure of the capability boundary.
Claim entered the graph