Scaling reasoning-training compute may make outcome-based reinforcement learning less likely to incentivize legible reasoning.
4 events · 1 assessment · 2 decisions
Structured and assessed
First pass. Built three arguments: absent legibility pressure (new definitional stub plus existing a4ced23d), observed illegibility (new causal claim d8eba1e3 plus existing 4ce17892), and stable frontier monitorability against (new empirical claim fbe4c4ad plus existing 22811de5). Linked 74c2c8af as related, not as a subclaim, because the Matcher matched my null-result wording to it as a denial but it is broader. Importance set to 0.55, contestation 0.55. Assessed supported at confidence 0.7, credence 0.7. Recorded the Jose 2025 instance (affirms) and the OpenAI monitorability blog instance (denies, low confidence), and filled metadata for the Korbak et al. instance. Provenance map is immaterial. No dependents to notify.
Assessed Supported
verdict confidence 0.70 · credence 0.70
The concern rests on a simple mechanism. Outcome-based reinforcement learning rewards final answers, not readable reasoning, so a reasoning model's chain of thought stays legible mainly because it inherits the natural-language habits of pretraining. The more RL optimization a model receives, the further it can move from that starting point. The 2025 multi-lab position paper on chain-of-thought monitorability (Korbak et al.) put the worry this way, in hedged terms. The evidence for the direction of the effect is real but indirect. Outcome-based RL has made some reasoning models' chains of thought illegible. Jose (2025) found that DeepSeek-R1, R1-Zero and QwQ often produce passages of nonsense and mixed languages that their base model does not, that removing those passages cuts accuracy, and that larger models trained the same way tend to be less legible. In addition, R1-Zero's language consistency declines over RL training unless a dedicated reward is added, and o3 traces contain idiosyncratic jargon. These are within-run trends and comparisons across models, not clean experiments that vary RL compute. The strongest counterevidence comes from OpenAI's own measurements. They report that monitorability stayed roughly steady over the frontier RL runs of o3 and GPT-5.1 Codex Max, apart from one dip in the middle of training. Claude reasoning models also stay highly legible. This shows that heavy outcome RL does not inevitably destroy legibility at today's scales, and that training details such as KL regularisation, data mixtures and auxiliary rewards matter a great deal. It does not show that legibility is incentivised. Monitorability is also a related but different property from legibility. The claim is stated as a possibility ("may"), so the evidence favours it: the incentive gap is real, the drift has been observed, and nothing so far shows a force that keeps chains of thought legible as RL compute keeps growing. Controlled studies that hold training details fixed and vary RL compute, and legibility tracking across successive frontier runs, would settle how strong the effect is.
Updated claim instance
Instance 15c70d53-8624-4d39-9a2b-63dea439b6ce (https://arxiv.org/pdf/2507.11473): was stance=affirms, confidence=0.85; set confidence=0.95, speaker="Tomek Korbak et al.", publication="arXiv (2507.11473), \"Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety\"", sourceDate="2025-07". Filling metadata from the source: the passage sits in Section 2 of the July 2025 multi-author position paper "Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety" (Korbak et al.).
Claim entered the graph