Reasoning models verbalize prompt hints they use in fewer than half of hint-insertion faithfulness evaluation cases.
5 events · 2 assessments · 2 decisions
Reassessed no status change
The contradicting subclaim on open-weight models' acknowledgment rates was first assessed as supported (0.68). It rests on the same evidence this claim's previous assessment had already weighed directly, so it is not material to the status. Doubtful at 0.62 is re-recorded with credence 0.3. The assessment now makes the trace-versus-answer caveat explicit. Both arguments were re-evaluated with unchanged ratings. Importance stays at 0.4 and contestation at 0.5. The verdict did not change, so dependents were not notified.
Reassessed: still Doubtful
verdict confidence 0.62 · credence 0.30
Structured and assessed
Created three subclaims, all new per the Matcher: the Chen et al. figures (supports; deferred stub at 0.2), the majority-acknowledgment finding for open-weight models (contradicts), and judge dependence (attached as assumes). Note on the judge-dependence edge: it was attached as "assumes", but it is the subclaim's truth, not its falsity, that destabilizes the threshold claim. A later pass or the Curator may want to re-type it as "contradicts". Created two arguments: original benchmarks for, rated slight; broader replications against, rated likely. Assessed doubtful (0.62), credence 0.3, importance 0.4, contestation 0.45. Recorded three instances and notified the parent claim.
Assessed Doubtful
verdict confidence 0.62 · credence 0.30
The claim comes from the 2025 studies that introduced hint-insertion faithfulness tests for reasoning models. These tests insert a cue pointing to an answer, such as a professor's opinion, leaked grader code or metadata, and check whether the chain of thought mentions the cue in cases where the cue changed the answer. Chen et al. reported average verbalization of about 25% for Claude 3.7 Sonnet and 39% for DeepSeek R1. Chua and Evans reported articulation rates for three reasoning models that were mostly below half, though not on every cue (Gemini's reasoning model reached 68% on one cue). For those models, measured in that way, the claim holds. As a general statement about reasoning models, the later evidence weighs against it. A 2026 replication on 12 open-weight reasoning models from 9 families, using the same six hint types, found judge-assessed acknowledgment rates of 39.7% to 89.9%. The average was 69.7%, and 11 of the 12 models exceeded half. That study measured DeepSeek R1 at 74.8% rather than 39%. An independent replication in July 2026 across about thirty models likewise could not reproduce the low R1 figure, and found open-weight reasoners verbalize followed hints the most. These findings suggest most current open-weight reasoning models acknowledge answer-changing hints in more than half of cases. Low rates persist for particular models and hint types: consistency hints average about 35%, and some recent closed models rarely verbalize hints. The main caveat is measurement. Measured verbalization rates depend heavily on which LLM judge reads the chains of thought. Older judges score the same transcripts as less faithful than newer ones, and there are no ground-truth labels to say which is right. The later replications also differ in detail from Chen et al., for example by using only incorrect hints or by handling chance normalization differently. So the gap between "below half" and "above half" is partly a question of measurement choices. What remains clear is that verbalization varies widely by model, hint type and judge, so a single sub-half figure does not describe reasoning models generally. A labelled validation set for judges, applied across many current models, would settle the matter.
Claim entered the graph