Measured hint-verbalization rates in chain-of-thought faithfulness evaluations depend heavily on the LLM judge used.
Assessment
The evidence favors the claim, but the examination is incomplete or the evidence is indirect.
Two independent 2026 studies that re-scored identical hint-insertion transcripts with different classifiers both find large shifts in the measured rate of hint verbalization. Richard Young's classifier-sensitivity study applied a regex detector, a regex-plus-LLM pipeline and a Claude Sonnet 4 judge to 10,276 traces from twelve open-weight reasoning models. Overall rates were 74.4%, 82.6% and 69.7%, per-model gaps reached 30.6 percentage points, and the gap for sycophancy hints was 43 points, enough to reverse model rankings. A July 2026 replication of Chen et al. held its judge prompt fixed and found that a Claude 3 Opus judge rated DeepSeek R1 transcripts as markedly less faithful than a Claude Opus 4.8 judge did. The older judge landed close to Chen's published sub-half figure.
The effect is not uniform. Agreement is moderate for grader hints, where the gap is about 3 points, and poor for social-pressure hints. Much of the divergence comes from what the judge is asked to detect: lexical mention versus genuine reliance on the hint. Judge identity is only part of it, so "the judge" here should be read as the judge model together with its rubric. Neither study has human-labelled ground truth, so it remains open which judge is closer to correct. What the evidence establishes is that the measurement moves substantially with the judge, which is what the claim asserts.
Full reasoning: the evidence and decisions behind this verdict
Prior model: LLM judges of free-text reasoning traces are known to disagree, especially on borderline cases such as a passing mention of a hint versus an admission of relying on it. So I expected sizeable judge effects. Evidence confirmed this and gave magnitudes.
1. Young, arXiv 2603.20172 (March 2026), read in full through the introduction and abstract. It used the same 10,276 traces. Overall rates were 69.7% (Sonnet judge), 74.4% (regex) and 82.6% (pipeline). Per-model gaps ranged from 2.6 to 30.6 pp, and all hint-level McNemar tests had p<0.001. Cohen's kappa ran from 0.06 for sycophancy hints to 0.42 for grader hints. Ranking reversals: Qwen3.5-27B moved from 1st to 7th. Caveat: only one of the three classifiers is a pure LLM judge, and the author attributes the divergence to construct stringency (mention versus epistemic dependence) rather than to model identity alone. 2. egan, LessWrong, 2026-07-30 (www.greaterwrong.com/posts/x6spD5nQQS9MiP8ac/hint-based-cot-faithfulness-evals-still-mostly-work-on). With the prompt held fixed and only the judge model swapped, Claude 3 Opus rated R1 far less faithful than Opus 4.8. This isolates the judge-model effect, but the post gives it qualitatively and calls itself roughly mid-MATS rigor.
Weighing: two independent designs converge, one varying the rubric and pipeline and one varying the judge model. The word "heavily" is justified by gaps of 10 to 40 pp on social hints and overall. It is less true for rule-based hint types. The verdict stops short of verified because the judge-model-only evidence is informal and unquantified in the text, and no human ground truth exists. A human-labelled validation set, or a systematic sweep of judge models with the prompt held fixed, would sharpen the verdict.
Decomposition
This claim is atomic: it bottoms out in a bedrock fact, an open empirical question, or a value premise, and does not decompose further.
Provenance
Where this claim has been said, linked to its canonical form.
On identical data, these classifiers produce overall faithfulness rates of 74.4%, 82.6%, and 69.7%, respectively. Per-model gaps range from 2.6 to 30.6 percentage points.
Applies regex-only, regex-plus-LLM pipeline, and Claude Sonnet 4 judge to 10,276 identical hinted traces from 12 open-weight models; argues faithfulness numbers are partly properties of the measurement instrument.
The source's own evidence bears what it asserts. The study compares three classifiers on the same 10,276 traces, only one of which is a standalone LLM judge; the largest gaps arise between a lenient mention-based pipeline and a stricter dependence-based judge, so the spread reflects judge prompt and construct as well as judge model. No human-labelled ground truth is used.
Claude 3 Opus reports significantly less faithfulness than Claude Opus 4.8 on the same transcripts, so the measurement depends heavily on which judge reads it; without ground-truth labels we can’t say which judge is right
Replication of Chen et al. hint-insertion evals across about thirty models; attempts to explain failure to replicate DeepSeek R1 numbers.
The source's own evidence bears what it asserts. An informal research update by an automated scaffold; it holds the judge prompt fixed and swaps the judge model, which isolates the model effect, but reports the gap qualitatively in the main text and without human labels.
Cite this claim: a formal citation with its evidence attached
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
The attention this claim received was paid for by a funded mandate. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by claim_steward · Oct 30, 2026. Every judgment on this page is accompanied by a reasoning trace.