In synthetic tasks with learned atomic skills, outcome-reward RL, but not rejection fine-tuning, generalizes to deeper unseen compositions.
3 events · 1 assessment · 1 decision
Assessed first pass
Kept atomic: a narrow empirical finding resting on one primary study plus reported replications; no named arguments needed. Importance 0.35, contestation 0.4. Read arXiv abstract of 2509.25123, recorded it as an affirming instance. Assessed supported (0.7, credence 0.75), noting the offline-RFT vs on-policy confound. Source map not written: single primary instance, provenance unremarkable. No dependents to notify beyond parent, which seeded this claim; assessment matches the seed.
Assessed Supported
verdict confidence 0.70 · credence 0.75
The claim rests mainly on Yuan et al., "From f(x) and g(x) to f(g(x))" (arXiv 2509.25123, ICLR 2026). In a synthetic setting where a model (Llama-3.1-8B-Instruct) has first learned atomic string-transformation functions, RL with an outcome reward on two-function compositions raised accuracy on unseen compositions of three and four functions well above baseline, while next-token training on the same data (rejection fine-tuning) produced essentially no such gain. Later synthetic studies (Cheng et al., arXiv 2512.01970; Kong et al., arXiv 2606.18089) are reported to show the same qualitative pattern in other settings. The finding is well supported within its scope, which the claim states: synthetic tasks where the atomic skills were learned beforehand. The main open question is attribution. The rejection fine-tuning baseline is offline, trained on filtered samples, whereas RL trains on-policy on the model's own evolving outputs and also learns from failures. The gap may therefore owe as much to on-policy data and negative signal as to the outcome reward as such, and an online rejection-sampling baseline would help separate these. The absolute levels of deeper-composition accuracy are also modest (tens of percent), so "generalizes" means substantial partial transfer, not mastery. Whether the pattern holds outside synthetic tasks is a separate question the claim does not assert.
Claim entered the graph