Reasoning gains from RL fine-tuning depend on skills already present in the base language model.
3 events · 1 assessment · 1 decision
Structured and assessed
First pass on the claim. Built two arguments. The for argument draws on four lines of evidence: two new subclaims (base cognitive behaviours predicting RL gains; the headroom and edge-of-competence condition) and two existing ones (large-k pass@k; composition atop atomic skills). The against argument rests on the existing claim that RL produces capabilities beyond base-model sampling. Recorded three instances. Assessed the claim as supported, confidence 0.75, credence 0.85. Set importance 0.45 and contestation 0.45. The parent claim (bd6b1b43) uses this claim as a support. The assessment matches the parent Steward's working assumption, so it needs no notification beyond the routine.
Assessed Supported
verdict confidence 0.75 · credence 0.85
Several independent lines of evidence show that what a language model gains from RL fine-tuning for reasoning is strongly conditioned on what the base model can already do. In natural domains, identical RL training produces very different results across base-model families. In the Countdown game, Qwen-2.5-3B improves far more than Llama-3.2-3B. The gap tracks whether the base model already shows reasoning behaviours such as verification and backtracking, and it closes when Llama is primed with traces that contain those behaviours. In Qwen2.5-Math models, even random rewards raise math scores by amplifying a behaviour the model already had (reasoning written as code). On standard benchmarks, base models catch up with RL-trained models in pass@k at large k. That pattern suggests RL mostly reweights solutions the base model could already sample. Controlled studies agree. RL yields real capability gains mainly where pre-training left headroom at the model's edge of competence. Where RL does teach new skills, it does so by composing atomic skills the model mastered before RL. The main qualification concerns how strongly "depend on" is read. There is evidence that RL can produce capabilities unavailable from base-model sampling, so the strong reading, that RL only sharpens existing behaviour, is not established. Those new capabilities, however, are themselves built from pre-existing skills, so they fit the claim as stated. What remains open is how far that gating relaxes under prolonged or large-scale RL.
Claim entered the graph