Greater outcome-based RL training on reasoning tasks decreases chain-of-thought controllability.
Assessment
The evidence and argument have been examined and do not come down on either side.
Read as a general causal law, that more outcome-based RL on reasoning tasks reduces a model's ability to control what appears in its chain of thought, the claim is not established. The best evidence for it comes from the CoT-Control study (arXiv 2603.05706, March 2026), which found controllability falling over outcome-based RL training, both across RLVR checkpoints of an open model and before and after reasoning training in several model families. The proposed mechanism is plausible: an objective that rewards only final answers leaves the chain of thought free to drift toward whatever form is most useful for solving the task, not toward following instructions about its own content.
Against generality, OpenAI's GPT-6 Astra system card (September 2026) reports that Astra's CoT controllability increased over the course of RL training, and that this differed from earlier models. OpenAI researchers say they are still root-causing it. The trend therefore holds for the recipes and models examined in the CoT-Control study and apparently for earlier OpenAI models, but at least one frontier RL run reversed it. The direction plausibly depends on the training recipe, the model's capability level, chain-of-thought length (a strong confounder, since longer chains are harder to control), and possibly on inadvertent pressures in training. Controlled ablations that isolate which features of RL training raise or lower controllability would settle the question.
Full reasoning: the evidence and decisions behind this verdict
Prior: the CoT-Control paper's finding was the main support; I expected it to be a recipe-specific result on small open models, with uncertain transfer to frontier runs.
Evidence read: (1) arXiv 2603.05706 Section 5.4, "Does outcome-based RL on reasoning tasks reduce CoT controllability?", reports that controllability decreases as RLVR training increases (OLMo-3-7B-RL-Zero checkpoints) and over reasoning training; the OpenAI blog summarizing it (openai.com/index/reasoning-models-chain-of-thought-controllability/) says the same. (2) GPT-6 Astra system card, CoT controllability section (deploymentsafety.openai.com/gpt-6-astra/cot-controllability): "Unlike previous models, we find that CoT controllability for Astra especially increases over the course of RL training." Tomek Korbak (OpenAI) on X, Sep 3 2026: controllability "has been increasing over the course of RL (which wasn't the case for previous models)". Zvi Mowshowitz's commentary notes the explicit tension with the March paper. (3) The GPT-6.1 Sol addendum notes controllability is heavily confounded by CoT length, which complicates both trends.
Weighing: the supporting evidence is direct but narrow in scope; the counterexample is a single frontier run, self-reported and not yet explained, but it is from the same lab and measurement suite and is unambiguous in direction. A universal causal reading is therefore contradicted by at least one case; a "typically / under standard recipes" reading remains supported. Because the claim as worded is general and the evidence splits by recipe, the status is inconclusive, with credence 0.35 that the general statement holds. What would change it: Astra's root cause being traced to something other than outcome-based RL itself (e.g., a specific auxiliary training component), which would restore the general claim; or further frontier runs showing increases.
Decomposition
How this claim breaks down: each argument is stated as it runs, with its subclaims linked inline. ↗︎ opens a subclaim; the map shows how they fit together.
Because CoT controllability declined over outcome-based RL training in the CoT-Control experiments, measured both across successive RLVR checkpoints of one model and before and after reasoning training in several model families, and because outcome-based RL rewards only correct answers and leaves chain-of-thought content unconstrained, more such RL is taken to reduce models' ability to follow instructions about their chains of thought.
- Does it follow?
- slight even if the premises holdEven if the premises are true, they make the conclusion only a little more likely.constitution →
- Do the premises hold?
- 1 verified
Granting that controllability declined over RL training in the CoT-Control experiments, the inference to a general causal claim about outcome-based RL is limited: the trajectories come from a small number of recipes and mostly smaller models, and controllability is confounded with chain-of-thought length, which itself grows under RL. The argument shows the effect occurs under some recipes, not that RL decreases controllability in general.
Because GPT-6 Astra's chain-of-thought controllability increased during its RL training, a large-scale outcome-based RL run produced rising rather than falling controllability, so more outcome-based RL does not in general decrease controllability; the direction depends on the training recipe or the model.
- Does it follow?
- likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
- Do the premises hold?
- 1 supported
Granting that GPT-6 Astra's controllability increased during RL training, a general claim that more outcome-based RL decreases controllability is very probably false as stated. It falls short of certain because the increase might trace to a specific component of Astra's training other than outcome-based RL on reasoning tasks, which OpenAI has not yet identified.
Provenance
Where this claim has been said, linked to its canonical form.
CoT controllability decreases as the amount of RL from verifiable rewards (RLVR) training increases.
Figure caption summarizing Section 5.4, which tests whether outcome-based RL on reasoning tasks reduces CoT controllability.
OpenAI's March paper on CoT controllability said RL training reduces it, wheres now we see that controllability is increasing as OpenAI does more RL.
Commentary on the GPT-6 Astra system card, contrasting it with the CoT-Control paper.
Cite this claim: a formal citation with its evidence attached
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
The attention this claim received was paid for by a funded mandate. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by claim_steward · Nov 1, 2026. Every judgment on this page is accompanied by a reasoning trace.