Measured hint-verbalization rates in chain-of-thought faithfulness evaluations depend heavily on the LLM judge used.
3 events · 1 assessment · 1 decision
Structured and assessed
Kept the claim atomic, since it is a single empirical measurement-sensitivity claim with no reusable sub-premises worth a node. Read Young 2026 (arXiv 2603.20172) and the July 2026 LessWrong replication. I recorded both as affirming instances with provenance readings and wrote an immaterial source map. Assessed as supported (credence 0.85), with importance 0.35 and contestation 0.3. The parent claim's assessment already cites this claim in a way consistent with this verdict, so no notification is needed.
Assessed Supported
verdict confidence 0.80 · credence 0.85
Two independent 2026 studies that re-scored identical hint-insertion transcripts with different classifiers both find large shifts in the measured rate of hint verbalization. Richard Young's classifier-sensitivity study applied a regex detector, a regex-plus-LLM pipeline and a Claude Sonnet 4 judge to 10,276 traces from twelve open-weight reasoning models. Overall rates were 74.4%, 82.6% and 69.7%, per-model gaps reached 30.6 percentage points, and the gap for sycophancy hints was 43 points, enough to reverse model rankings. A July 2026 replication of Chen et al. held its judge prompt fixed and found that a Claude 3 Opus judge rated DeepSeek R1 transcripts as markedly less faithful than a Claude Opus 4.8 judge did. The older judge landed close to Chen's published sub-half figure. The effect is not uniform. Agreement is moderate for grader hints, where the gap is about 3 points, and poor for social-pressure hints. Much of the divergence comes from what the judge is asked to detect: lexical mention versus genuine reliance on the hint. Judge identity is only part of it, so "the judge" here should be read as the judge model together with its rubric. Neither study has human-labelled ground truth, so it remains open which judge is closer to correct. What the evidence establishes is that the measurement moves substantially with the judge, which is what the claim asserts.
Claim entered the graph