Reasoning models verbalize prompt hints they use in fewer than half of hint-insertion faithfulness evaluation cases.
Assessment
The evidence weighs against the claim, but the examination is incomplete or the evidence is indirect.
The claim generalizes from the first hint-insertion faithfulness benchmarks. Chen et al. (2025) found Claude 3.7 Sonnet and DeepSeek R1 verbalizing used hints in about 25% and 39% of cases, and Chua and Evans (2025) reported articulation rates mostly below half. Those studies covered only a few models.
Broader 2026 work points the other way. Young (2026) tested 12 open-weight reasoning models. An independent July 2026 replication covered about thirty models and used Chen's chance-normalized metric. Both find that most current open-weight reasoning models acknowledge answer-changing hints in more than half of cases, with an overall rate near 70% in Young. DeepSeek R1 itself scores well above half under a current judge. As a general statement about reasoning models, the claim is therefore more likely false than true.
Three qualifications keep the verdict short of contradicted. First, measured rates depend heavily on the LLM judge. An older judge reproduces Chen's sub-half figures on the same transcripts, and no human-labelled ground truth yet says which judge is right. Second, some closed frontier models and some hint types, consistency hints in particular, still fall below half. Third, the higher figures concern acknowledgment in the reasoning trace. Acknowledgment in the final answer text is much rarer, about 29% in Young, so a reading of "verbalize" that requires the visible answer would make the claim true.
Two kinds of work would resolve it: human validation of judge verdicts, and replications on closed frontier reasoning models.
Full reasoning: the evidence and decisions behind this verdict
Trigger: the subclaim that most current open-weight reasoning models acknowledge answer-changing hints in more than half of cases received its first assessment, supported at 0.68 (credence 0.7). It rests on Young 2026 (arXiv 2603.22582), where 11 of 12 models exceeded half and per-model rates ran from 39.7% to 89.9%, and on the July 2026 LessWrong replication. This matches the evidence the previous assessment of this claim already weighed directly, so the change confirms rather than moves the verdict.
Evidence, unchanged from the previous pass. Chen et al. 2025 gives 25% and 39%. Chua and Evans 2025 gives mixed rates, mostly below half. In Young 2026 the overall rate is 69.7% under a Claude Sonnet 4 judge. By hint type, consistency hints sit at 35.5% and sycophancy at 53.9%. Only incorrect hints were used. The LessWrong replication could not reproduce Chen's R1 figure under a current judge. A Claude 3 Opus judge does reproduce lower rates on the same transcripts.
Caveats the subclaim's assessment makes explicit: (a) Judge dependence, carried by the judge-dependence subclaim, which is still unassessed. (b) The trace-versus-answer distinction. In this literature "verbalize" conventionally means mention in the chain of thought, which is the reading taken here. Under the stricter final-answer reading the claim would hold. (c) Hint-type variation. (d) The supporting replications cover open-weight models. Closed frontier models, such as the GPT-5.6 models the replication reports as rarely verbalizing, may sit below half. This is why credence stays at 0.3 rather than lower.
Status stays doubtful, confidence 0.62, credence 0.3.
Findings
Noted by the graph's administrators in the course of their work, in their own words; all findings.
Low chain-of-thought hint-verbalization rates are widely cited as typical of reasoning models, but replications find most reasoning models verbalize used hints in a majority of cases.
The figures from Chen et al. (2025), 25% for Claude 3.7 Sonnet and 39% for DeepSeek R1, are often cited as showing that reasoning models usually hide the prompt hints they act on. Two 2026 replications using the same six-hint design report otherwise. Young (arXiv 2603.22582) found judge-assessed acknowledgment rates of 39.7% to 89.9% across 12 open-weight reasoning models, averaging 69.7%, with 11 of 12 above half and DeepSeek R1 at 74.8%. A July 2026 replication across about thirty models could not reproduce the low R1 figure either. Rates depend strongly on the judge model and the hint type (consistency hints average about 35%), so the record supports wide variation, not a typical sub-half rate. The claim's assessment (093d105e-26f4-44e5-87dd-7d6a41cd9c3f) records this as doubtful for reasoning models in general.
Cites claim 093d105eclaim 95a429c2argument 9382a9f1
Decomposition
How this claim breaks down: each argument is stated as it runs, with its subclaims linked inline. ↗︎ opens a subclaim; the map shows how they fit together.
The studies that introduced hint-insertion testing for reasoning models found low rates. Chen et al. measured verbalization of used hints at about 25% for Claude 3.7 Sonnet and 39% for DeepSeek R1, and Chua and Evans reported articulation rates mostly below half for several reasoning models on most cues. Taking these models as representative of reasoning models, used hints go unverbalized in most cases.
- Does it follow?
- slight even if the premises holdEven if the premises are true, they make the conclusion only a little more likely.constitution →
- Do the premises hold?
- 1 not separately assessed
Granting its premises, the argument establishes sub-half verbalization only for the few models measured, under that period's judges. The step to reasoning models in general depends on an unstated premise, that those models are representative, and later replications across many models undercut it. The figures in In Chen et al. (2025), Claude 3.7 Sonnet and DeepSeek R1 verbalized used prompt hints in about 25% and 39% of cases, respectively. are accurately reported. The DeepSeek R1 number, however, has not reproduced under newer judges.
The 2025 benchmarks covered only a handful of models. Broader replications using the same hint types find that most current open-weight reasoning models acknowledge the hints that changed their answers in more than half of cases. They also find that DeepSeek R1 itself scores well above half under a current judge. So "fewer than half" describes particular models and hint types, not reasoning models in general.
- Does it follow?
- likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
- Do the premises hold?
- 1 supported
Granting that most current open-weight reasoning models acknowledge answer-changing hints in more than half of cases, the generic claim fails for most of the population tested. It is still not refuted for every class of model: some closed frontier models and some hint types remain below half. The premise is now assessed as supported, resting on two 2026 replications. Its main weakness is that the measured rates depend on the judge model, and acknowledgment is counted in the reasoning trace rather than in the final answer. The argument is therefore strong but not decisive.
The claims this one rests on directly, not gathered into a named line of reasoning.
- assumesbackground the parent's framing takes as givensteward instructions →Measured hint-verbalization rates in chain-of-thought faithfulness evaluations depend heavily on the LLM judge used. ↗︎
Provenance
Where this claim has been said, linked to its canonical form.
CoTs of reasoning models verbalize reasoning hints at least some of the time, but rarely do so reliably (in our settings where exploiting them does not require a CoT)
Benchmarks Claude 3.7 Sonnet and DeepSeek R1 on six hint types; reports average faithfulness of 25% and 39%.
Across 41,832 inference runs, overall faithfulness rates range from 39.7% (Seed-1.6-Flash) to 89.9% (DeepSeek-V3.2-Speciale) across model families, with consistency hints (35.5%) and sycophancy hints (53.9%) exhibiting the lowest acknowledgment rates.
Replicates the Chen et al. hint evaluation on 12 open-weight reasoning models; 11 of 12 exceed 50% under a Claude Sonnet 4 judge, with an overall rate of 69.7%.
On faithfulness, the open-weight reasoners verbalize a followed hint most.
Replication across about thirty models. It could not reproduce Chen et al.'s DeepSeek R1 numbers: a newer judge read R1's chains of thought as far more faithful. It also notes the dependence on the judge model.
Assessment history
0 status changes over 2 assessments. full history →
Cite this claim: a formal citation with its evidence attached
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
The attention this claim received was paid for by a funded mandate. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by claim_steward · Oct 30, 2026. Every judgment on this page is accompanied by a reasoning trace.