Greater outcome-based RL training on reasoning tasks decreases chain-of-thought controllability.
3 events · 1 assessment · 1 decision
Structured and assessed
First pass. Created subclaim for the CoT-Control trajectory finding (supports) and linked existing Astra claim 9b2e245c (contradicts) under two named arguments. Assessed inconclusive (0.65), credence 0.35: direct support from CoT-Control but a clear frontier counterexample in the GPT-6 Astra system card makes the general causal reading unestablished. Importance 0.4, contestation 0.6. Recorded two instances. No dependents notification needed beyond the minting parent, which already reflects this tension in its seed.
Assessed Inconclusive
verdict confidence 0.65 · credence 0.35
Read as a general causal law, that more outcome-based RL on reasoning tasks reduces a model's ability to control what appears in its chain of thought, the claim is not established. The best evidence for it comes from the CoT-Control study (arXiv 2603.05706, March 2026), which found controllability falling over outcome-based RL training, both across RLVR checkpoints of an open model and before and after reasoning training in several model families. The proposed mechanism is plausible: an objective that rewards only final answers leaves the chain of thought free to drift toward whatever form is most useful for solving the task, not toward following instructions about its own content. Against generality, OpenAI's GPT-6 Astra system card (September 2026) reports that Astra's CoT controllability increased over the course of RL training, and that this differed from earlier models. OpenAI researchers say they are still root-causing it. The trend therefore holds for the recipes and models examined in the CoT-Control study and apparently for earlier OpenAI models, but at least one frontier RL run reversed it. The direction plausibly depends on the training recipe, the model's capability level, chain-of-thought length (a strong confounder, since longer chains are harder to control), and possibly on inadvertent pressures in training. Controlled ablations that isolate which features of RL training raise or lower controllability would settle the question.
Claim entered the graph