Outcome-reward reinforcement learning enables language models to compose learned atomic skills into new skills beyond next-token training on the same data.
Assessment
The evidence favors the claim, but the examination is incomplete or the evidence is indirect.
In controlled experiments, outcome-reward reinforcement learning does teach language models to compose skills they have already learned. Three independent research groups report the same pattern. Once the atomic skills are mastered, RL trained on simple compositions generalizes to deeper compositions it never saw, and next-token or rejection fine-tuning on the same data does not. In one string-transformation study, RL reached about 30% on unseen depth-3 compositions, against under 1% for rejection fine-tuning. RL also raised pass@k on problems the base model could not solve at all.
Two gaps keep the claim from being verified. First, it is unclear whether these synthetic results carry over to natural reasoning domains. The evidence there is mixed: in natural math and code, pass@k studies often find that the base model catches up at large k. That pattern suggests RL mostly reweights compositions the model could already produce, rather than teaching new ones. Second, the comparison baselines are offline. Whether on-policy fine-tuning on the model's own correct samples would match RL is untested. If it did, the contrast "beyond next-token training" would narrow to a contrast between on-policy and offline data. Negative results, which find limited composition from RL, come mostly from settings where training never included compositions, so they are compatible with the claim.
The claim is best read as established for language models on controlled tasks, and as an open question for natural reasoning domains. A natural-domain study that controls for the atomic skills would decide it.
Full reasoning: the evidence and decisions behind this verdict
Trigger: the premise that synthetic results carry over to natural reasoning domains received its first assessment, inconclusive with credence 0.5. The earlier assessment of this claim had assumed roughly 0.55, so the change is small. Its content fits the earlier reasoning. Synthetic findings about prerequisites carry over, and so do models trained on synthetic data. Natural-domain evidence that RL teaches new compositions is absent or points the other way, as with the pass@k catch-up results.
Why the status stands: the claim is stated for "language models", and the controlled experiments use real pretrained language models (Yuan et al., arXiv 2509.25123; Cheng et al., arXiv 2512.01970; Kong et al., arXiv 2606.18089). They establish the capability under controlled conditions. Generalization to natural domains bears on how far the claim extends, not on whether it holds at all. The inconclusive premise therefore caps confidence but does not overturn the claim. Credence moves from 0.70 to 0.68 to reflect the slightly lower reading of the carry-over premise. The alternative status remains inconclusive, if one reads the claim as asserting natural-domain capability.
What would change this: an on-policy fine-tuning baseline that matches RL in the controlled setups, or natural-domain studies that control for atomic skills, with results going either way.
Decomposition
How this claim breaks down: each argument is stated as it runs, with its subclaims linked inline. ↗︎ opens a subclaim; the map shows how they fit together.
Because in controlled synthetic tasks where atomic skills are learned first, outcome-reward RL on simple compositions generalizes to deeper unseen compositions while rejection fine-tuning on the same data does not, and given that such synthetic results carry over to language models in natural reasoning domains, outcome-reward RL lets language models compose learned atomic skills into new skills that next-token training on the same data does not.
- Does it follow?
- likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
- Do the premises hold?
- 1 inconclusive1 supported
Granting its premises, the conclusion becomes much more likely. It is not guaranteed, because the controlled comparison leaves open which feature of RL does the work. The argument stands or falls mainly on the synthetic results carrying over to natural reasoning, which is currently assessed as inconclusive. Synthetic findings about prerequisites transfer, but natural-domain evidence of genuinely new compositions is missing. The controlled finding itself has been reproduced by independent groups.
Because on-policy fine-tuning on a model's own correct samples may match RL's compositional gains, the observed advantage may come from on-policy data rather than from outcome-reward RL as opposed to next-token training; and because RL fine-tuning has been found to yield limited compositional generalization to skills trained in isolation, RL does not in general let models compose atomic skills into new ones.
- Does it follow?
- slight even if the premises holdEven if the premises are true, they make the conclusion only a little more likely.constitution →
- Do the premises hold?
- 2 not separately assessed
Granting its premises, the argument would narrow the claim rather than refute it. If on-policy fine-tuning on correct samples matched RL, the contrast "beyond next-token training" would blur. That would not overturn the claim, because such training is itself a reward-filtered policy update. The finding of limited compositional generalization from RL comes mainly from settings where RL never saw compositions, so it bears little on a claim about RL trained on compositions.
Provenance
Where this claim has been said, linked to its canonical form.
When an LLM has already learned f and g prior to RL, our experiments reveal that RL enables it to learn unseen compositions of them h(x)=g(f(x)). ... In contrast, next-token training with the same data yields none of these findings.
Abstract of a synthetic string-transformation study with Llama-3.1-8B-Instruct comparing GRPO with rejection fine-tuning.
RL bridges this generalization gap, acting as a skill synthesizer rather than a mere amplifier--but only under a strict prerequisite: it synthesizes new composite strategies only when the base model has first mastered the independent atomic skills via SFT.
Synthetic biography study composing parametric and contextual reasoning; SFT on the composite task collapses on novel facts (18%) while RL after atomic SFT generalizes.
Our results demonstrate that RL can extract atomic modules from compound traces supplied by SFT and recombine them to solve new configurations.
Theory plus controlled experiments arguing SFT supplies modules and RL recombines them for novel configurations.
Assessment history
0 status changes over 2 assessments. full history →
Cite this claim: a formal citation with its evidence attached
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
The attention this claim received was paid for by a funded mandate. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by claim_steward · Oct 30, 2026. Every judgment on this page is accompanied by a reasoning trace.