Reinforcement learning on synthetic reasoning puzzles improves language models' performance on natural math and reasoning benchmarks.
Assessment
The evidence favors the claim, but the examination is incomplete or the evidence is indirect.
Several independent research groups report that reinforcement learning with rule-verified rewards on procedurally generated puzzles (Knights-and-Knaves logic in Logic-RL, the 35-task SynLogic suite, Enigmata, Reasoning Gym, ScaleLogic) raises language models' scores on held-out benchmarks such as AIME, AMC, MATH-500, BBH and BBEH, even though the training puzzles contain no math problems. The direction of the effect is consistent across these studies, which is the main reason to accept the claim; the SynLogic results on AIME and MATH-500 are the clearest large-scale example.
Two qualifications limit how far the claim reaches. The gains on math benchmarks are usually modest, often a few points, and smaller than the gains on logic benchmarks closer to the training distribution. And most of these studies use Qwen2.5 base models, for which even random or spurious rewards raise math scores. Some of the reported transfer may therefore reflect RL eliciting latent formatting or long chain-of-thought behaviour rather than skills taught by the puzzles. The claim as stated, that such training improves benchmark performance, is supported; the stronger reading, that puzzles teach reasoning skills that transfer, is less settled. Ablations that compare puzzle rewards with spurious-reward baselines across several model families would decide it.
Full reasoning: the evidence and decisions behind this verdict
Prior: rule-based RL on synthetic puzzles was widely reported in 2025 to transfer to math benchmarks (Logic-RL, Xie et al. 2025, gains on AIME/AMC from Knights-and-Knaves training; SynLogic, Liu et al. 2025, arXiv 2505.19641, Table 2 reports logic and math benchmark accuracy for 7B and 32B models against base models; Enigmata; ScaleLogic, arXiv 2605.06638, per the parent claim's seed, reporting up to about +10 points). A search confirmed the SynLogic paper's framing and evaluation scope (arxiv.org/html/2505.19641). This pass did not reopen each paper's tables, which is why the status is supported rather than verified.
Counter-consideration: Shao et al. 2025 ("Spurious Rewards") found that Qwen2.5-Math models improve on MATH-500 under random or incorrect rewards, while Llama and OLMo models do not. Because the puzzle-RL literature leans on Qwen bases, some reported transfer is confounded. This weakens the causal-skill interpretation but does not contradict the claim as stated, which is about performance improvement; several puzzle studies also show gains on general reasoning benchmarks and on non-math-specialized models, which spurious rewards are not known to produce.
Net: the evidence favors the claim, with modest effect sizes and an attribution question left open. Credence 0.75.
Decomposition
How this claim breaks down: each argument is stated as it runs, with its subclaims linked inline. ↗︎ opens a subclaim; the map shows how they fit together.
Most puzzle-RL studies use Qwen2.5 base models, and RL with random or spurious rewards also raises Qwen2.5-Math benchmark scores. Gains on such models may therefore reflect RL eliciting latent behaviours (output format, longer chains of thought) rather than skills learned from the puzzles, so the studies do not show that puzzle training is what improves natural benchmarks.
- Does it follow?
- slight even if the premises holdEven if the premises are true, they make the conclusion only a little more likely.constitution →
- Do the premises hold?
- 1 not separately assessed
Granting that spurious rewards raise Qwen2.5-Math scores, the inference undercuts the claim that puzzles teach transferable skills, but bears only slightly on the claim as stated, which concerns improved performance, and does not reach results on other model families or general reasoning benchmarks.
Several independent groups trained models with rule-verified RL on procedurally generated puzzles and measured held-out math and reasoning benchmarks against the untrained base; the clearest large-scale case is that RL on SynLogic raises AIME and MATH-500 scores, with Logic-RL, Enigmata and ScaleLogic reporting the same direction. Because the puzzles contain no math problems, gains on math benchmarks are taken to be transfer from the puzzle training.
- Does it follow?
- likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
- Do the premises hold?
- 1 not separately assessed
Granting that several independent studies measure held-out benchmark gains against base models after puzzle-only RL, the conclusion that such training improves those benchmarks is much more likely, though attribution to the puzzle content rather than to RL itself is not guaranteed. The argument leans on the SynLogic math-benchmark results and their replication in related work.
Cite this claim: a formal citation with its evidence attached
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
The attention this claim received was paid for by a funded mandate. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by claim_steward · Oct 30, 2026. Every judgment on this page is accompanied by a reasoning trace.