Reasoning models' chain-of-thought legibility declines as question difficulty increases.
3 events · 1 assessment · 1 decision
Structured and assessed
Kept the claim atomic. Its only evidence base is one study, and splitting by model family or adding a length-confound subclaim would not pass the reuse bar. The confound is handled in the prose: the source normalizes for length. Read Jose 2025 in full on the relevant sections and recorded one instance with a reading (overstates, because of the caption/text inconsistency and the Claude exception) and a material source map. Importance set to 0.4 and contestation to 0.3. Assessed supported (confidence 0.65, credence 0.7). This is a first assessment, and the parent's seed (0.65) is consistent with it, so the parent was not notified.
Assessed Supported
verdict confidence 0.65 · credence 0.70
The direct evidence is a single study, Jose (2025, arXiv 2510.27338, NeurIPS 2025). It scored chain-of-thought legibility for 14 models on GPQA-Diamond questions, with the answer choices removed and GPT-4o as the grader. Reasoning models trained with outcome-based reinforcement learning that already produce illegible text (DeepSeek R1, R1-Zero, QwQ, and to a lesser degree Qwen3 and the R1 distills) were rated less legible on questions classed as hard. R1 showed about a 23% relative rise in illegibility from easy to hard questions. Harder questions draw longer chains of thought, and longer chains are less legible, but the effect survived normalizing for length. The study's authors note that it was stronger without that normalization. The claim holds as a tendency within that class of models, not as a property of reasoning models in general. The paper's text reports that Claude models, which are almost fully legible, show no such decline. Kimi K2 also shows almost no illegibility. The evidence has further limits. It comes from one benchmark, with a coarse two- or three-level hardness split. Legibility was judged by an LLM rather than by humans, though a second grader agreed well. No significance tests are given for the difficulty comparison. The paper is also inconsistent about which model's correctness defined hardness: the text names Claude 3.7 Sonnet and a figure caption names R1. A plausible mechanism fits the result, since harder problems need more reasoning steps and more compressed "bridging" tokens. A second study pointing the same way found that chain-of-thought faithfulness falls on harder tasks, though faithfulness is a different property. A replication on other benchmarks, with human raters and a continuous difficulty measure, would settle how general the effect is.
Claim entered the graph