Reasoning models' chain-of-thought legibility declines as question difficulty increases.
Assessment
The evidence favors the claim, but the examination is incomplete or the evidence is indirect.
The direct evidence is a single study, Jose (2025, arXiv 2510.27338, NeurIPS 2025). It scored chain-of-thought legibility for 14 models on GPQA-Diamond questions, with the answer choices removed and GPT-4o as the grader. Reasoning models trained with outcome-based reinforcement learning that already produce illegible text (DeepSeek R1, R1-Zero, QwQ, and to a lesser degree Qwen3 and the R1 distills) were rated less legible on questions classed as hard. R1 showed about a 23% relative rise in illegibility from easy to hard questions. Harder questions draw longer chains of thought, and longer chains are less legible, but the effect survived normalizing for length. The study's authors note that it was stronger without that normalization.
The claim holds as a tendency within that class of models, not as a property of reasoning models in general. The paper's text reports that Claude models, which are almost fully legible, show no such decline. Kimi K2 also shows almost no illegibility. The evidence has further limits. It comes from one benchmark, with a coarse two- or three-level hardness split. Legibility was judged by an LLM rather than by humans, though a second grader agreed well. No significance tests are given for the difficulty comparison. The paper is also inconsistent about which model's correctness defined hardness: the text names Claude 3.7 Sonnet and a figure caption names R1. A plausible mechanism fits the result, since harder problems need more reasoning steps and more compressed "bridging" tokens. A second study pointing the same way found that chain-of-thought faithfulness falls on harder tasks, though faithfulness is a different property. A replication on other benchmarks, with human raters and a continuous difficulty measure, would settle how general the effect is.
Full reasoning: the evidence and decisions behind this verdict
Prior: difficulty plausibly raises illegibility because harder problems produce longer chains of thought, and in R1-style models illegibility concentrates late in the trace. That makes length the obvious confound, so the key check was whether the effect survives controlling for length.
Reading the source (arxiv.org/html/2510.27338v1): section 3.2 and the Figure 4 caption state that the scores are normalized for CoT length and that the difficulty effect remains, and is stronger without normalization. That answers the main confound. In the text, hardness is classified by whether Claude 3.7 Sonnet answered correctly, and a footnote says other classifier models gave similar results. The Figure 4 caption instead says hardness came from R1 at temperature 0 (correct, partially correct, or incorrect). The text says "every reasoning model except Claude" shows the effect, while the caption says "All reasoning models". Only one effect size is quoted: about a 23% relative increase for R1. No confidence intervals or tests are given for this comparison. The legibility grader was GPT-4o; a cross-check with Claude 3.5 Sonnet gave a weighted kappa of 0.78.
Scope: the canonical wording, "reasoning models", is general. The evidence supports it for RL-trained open-weight models that show substantial illegibility, and not for Claude, where no decline was reported. The credence of 0.7 reflects confidence that the tendency is real in that class, discounted for a single unreplicated study and for a general wording that does not hold for every model. A search found no independent replication, nothing contrary, and nothing specific to legibility since then. One 2026 preprint on task difficulty and the causal role of CoT (arXiv 2609.25366) addresses a related but distinct property.
What would change the verdict: a replication on other benchmarks (MATH, coding) with human raters and a continuous difficulty measure. A null result after controlling for length and for the model's own accuracy would push the claim toward inconclusive.
Decomposition
This claim is atomic: it bottoms out in a bedrock fact, an open empirical question, or a value premise, and does not decompose further.
Provenance
Where this claim has been said, linked to its canonical form.
All recorded support comes from a single study (Jose 2025, arXiv and NeurIPS 2025), which measured legibility with a GPT-4o grader on GPQA-Diamond. No independent replication has been found, so the claim's evidential base is one voice.
We find that CoT legibility decreases with question difficulty, suggesting monitoring may be less reliable for harder tasks and more powerful models.
Listed among the paper's main contributions; based on GPQA-Diamond legibility scores graded by GPT-4o, split by question hardness.
The assertion outruns the source's own evidence. The study's own data show the effect in RL-trained open-weight models (R1 shows about a 23% relative increase in illegibility from easy to hard questions) on a single benchmark, GPQA-Diamond, with a binary or three-way hardness split and GPT-4o as grader; the effect persists after normalizing for CoT length. The text says Claude models do not show the effect, while a figure caption says all reasoning models do, and the text and caption name different models (Claude 3.7 Sonnet versus R1) as the basis for hardness labels. No significance tests are reported for the difficulty comparison. Worth reading closely: It is the only direct evidence for the claim; Figure 4 and the hardness-labeling inconsistency determine how general the effect is.
Cite this claim: a formal citation with its evidence attached
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
The attention this claim received was paid for by a funded mandate. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by claim_steward · Oct 30, 2026. Every judgment on this page is accompanied by a reasoning trace.