Outcome-reward reinforcement learning enables language models to compose learned atomic skills into new skills beyond next-token training on the same data.
5 events · 2 assessments · 2 decisions
Reassessed minor
The subclaim on synthetic-to-natural carry-over received its first assessment, inconclusive at 0.5, close to the assumed 0.55. Not material: the status stays supported at confidence 0.65, and credence moves from 0.70 to 0.68. Argument evaluations refreshed to reflect the premise's assessment. No dependents need notifying.
Reassessed: still Supported
verdict confidence 0.65 · credence 0.68
Structured and assessed
Decomposed into two arguments: for (controlled composition finding plus an assumed external-validity premise) and against (on-policy SFT alternative; limited-composition findings). All four subclaims are new per the Matcher and were seeded. Read Yuan et al. v3 in full through the setup, plus the abstracts of two independent replications. Assessed supported, 0.65 / 0.70. Importance 0.4, contestation 0.45. Recorded three affirming instances. Wrote an immaterial provenance map. Notified the parent.
Assessed Supported
verdict confidence 0.65 · credence 0.70
Controlled experiments support the claim. The best evidence comes from synthetic tasks in which a model first learns atomic skills and then trains on simple compositions of them. Yuan et al. (ICLR 2026) used string transformations with Llama-3.1-8B-Instruct. Outcome-reward RL on two-function compositions raised accuracy on unseen three-function compositions from near zero to about 30%, and on four-function compositions to about 15%. It also transferred to a different task, Countdown arithmetic. Rejection fine-tuning, which is next-token training on correct solutions to the same problems, gained less than 1%. Two independent groups found the same pattern in other synthetic settings: Cheng et al. with composed retrieval and in-context reasoning, and Kong et al. with modular reasoning traces. So the controlled finding has been reproduced across independent groups. Two qualifications limit the general claim. First, the evidence is synthetic. Whether these results carry over to natural reasoning domains is largely untested, because there atomic skills are not cleanly separable. Second, the baseline is offline rejection fine-tuning. The comparison therefore does not cleanly separate the reward signal from the fact that RL trains on the model's own current samples. It is open whether on-policy fine-tuning on the model's own correct samples would close the gap. Studies finding that RL composes skills poorly trained RL only on isolated skills. The composition studies reproduce that failure too, so those findings mark a condition for the effect rather than a contradiction: RL has to be trained on compositions. The effect also requires that the atomic skills already be learned.
Claim entered the graph