The argument from measured unfaithfulness
The argument
Natural language is not the same as faithfulness. Today's reasoning models often omit from their chains of thought the hints or reward hacks that drive their answers, and more capable future models may be able to deliberately obscure their reasoning. Systems capable enough to automate AI research, trained with heavy outcome-based reinforcement learning, may therefore write natural-language chains that are not mostly faithful.
Does it work?
- Does it follow?
- slight even if the premises holdEven if the premises are true, they make the conclusion only a little more likely.constitution →
- Do the premises hold?
- 2 supported
Granting its premises, the argument shows only that faithfulness is not guaranteed. The omissions found in hint-omission studies concern cues that can be used without much reasoning, and acknowledgment rates near 70% are compatible with "mostly". That future models may be able to obscure reasoning is a capability claim; it does not show that the first automators will in fact do so. The argument weighs more heavily against a strict reading of "faithful" than against a monitorability reading.
Premises
- SupportedThe evidence favors the claim, but the examination is incomplete or the evidence is indirect.constitution →Reasoning models often omit from chain of thought hints or reward hacks that drive their answers.
- SupportedThe evidence favors the claim, but the examination is incomplete or the evidence is indirect.constitution →Future AI models will be able to deliberately obscure their reasoning to evade chain-of-thought monitoring.
Where it is made
Passages where sources make, criticize or mention this argument. They record where it appears, not how strong it is, and none has precedence over another.
No source is recorded as making it.