trailUnconstrained chain-of-thought from pure reinforcement learning can develop reasoning beyond human reasoning priors.
Scaling reasoning-training compute may make outcome-based reinforcement learning less likely to incentivize legible reasoning.
The argument from the cost of readability constraints· for
Outcome-based reinforcement learning can make some reasoning models' chains of thought illegible.
DeepSeek-R1-Zero produces chain-of-thought responses with poor readability and English-Chinese language mixing.
In DeepSeek's RL ablation on DeepSeek-R1-Distill-Qwen-7B, chain-of-thought language consistency declines without an LC reward but stays stable with one.
Some open-weight reasoning models, including DeepSeek-R1 and QwQ, produce illegible chain-of-thought unlike their non-RL base models.
A language-consistency reward improves chain-of-thought readability while slightly reducing model performance.
A language-consistency reward during RL training keeps a model's chain of thought in the target language.
Language mixing reduces human readability of reasoning models' chains of thought.
Enforcing monolingual chain-of-thought reasoning lowers reasoning accuracy in language models.
The argument from capability expansion beyond the base model· for
Reinforcement learning with verifiable rewards can produce reasoning capabilities unavailable from base-model sampling.
Prolonged reinforcement learning training can enable models to solve tasks their base models cannot solve, regardless of sample count.
↓
↓
↓
Outcome-reward reinforcement learning enables language models to compose learned atomic skills into new skills beyond next-token training on the same data.
↑
⇄
◌
At high k, base models match or exceed RLVR-trained models in pass@k on math, code, and visual reasoning benchmarks.
Power-sharpened sampling from an untrained base model matches RL-posttrained models' single-shot reasoning performance.
+4
claim · empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →claim page ↗︎
Unconstrained chain-of-thought from pure reinforcement learning can develop reasoning beyond human reasoning priors.
⇄InconclusiveThe evidence and argument have been examined and do not come down on either side.constitution →credence 0.42Credence, from 0 to 1: the Steward's probability that the claim, as stated, is true. Stated only where a single number is an honest summary; normative and evaluative claims usually carry none.constitution →
1depended on by
this rests on ↓
claimA box is a claim: a single proposition the graph assesses, with its own page and map. Click any claim to centre the map on it.constitution →quantityA box with a rule beneath its text is a quantity: a value the graph estimates rather than a proposition it affirms or denies. Its estimate stands where a claim's status would.constitution →questionA rounded box is a question: answered rather than affirmed or denied, with its candidate answers, subquestions and presuppositions beneath it. Its mark says how well answered it is.constitution →argumentA pill is an argument: one line of reasoning stating how the claims beneath it combine to bear on the claim above it, for or against. Arguments are not destinations; click their claims to explore.constitution →✓verifiedThe evidence, examined directly, establishes the claim.constitution →↑supportedThe evidence favors the claim, but the examination is incomplete or the evidence is indirect.constitution →⇄inconclusiveThe evidence and argument have been examined and do not come down on either side.constitution →↓doubtfulThe evidence weighs against the claim, but the examination is incomplete or the evidence is indirect.constitution →✕contradictedThe evidence, examined directly, weighs against the claim.constitution →?unknownInsufficient information to assess.constitution →◌unassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →verified factunsettledvalue premisetheoremprizeA double ring marks a claim with a live prize: an amount in owls offered for a machine-checked proof or disproof of its published formal statement. The amount is in the preview and on the claim page; a prize changes nothing about how the claim is assessed or how important it is judged to be.prize rules →⊢machine-checked⊢ marks a claim whose published formal statement has a machine-checked proof or disproof. The checker confirms the proof; the verdict beside it is still the steward's judgment of the claim as worded.prize rules →supportsthis provides evidence for the parentsteward instructions →contradictsthis argues against the parentsteward instructions →assumesbackground the parent's framing takes as givensteward instructions →requiresa load-bearing premise: the parent is false without itsteward instructions →answersa candidate answer to the questionconstitution →Fig. Detail falls off with distance; every claim is an address.