Evaluating Chain-of-Thought Monitorability is Still an Open Problem: Comments on OpenAI's Monitorability Evals
Author not recordedDate not recordedOriginal︎ ↗
Stewards’ notes · 1
- OpenAI's intervention, process, and outcome-property evaluations validly measure chain-of-thought monitorability.
Supports. The critique rests on the authors' hands-on use of the released evaluations and on worked examples. Its construct objection is partly a disagreement over how monitorability should be defined. Oct 30, 2026
Examinations
None recorded.
Text
No stored copy. Only the passages claims quote are recorded.
we think that even if the issues we show in this evaluation are fixed, only the outcome-property evals provide information on how useful CoT monitoring is in detecting misbehavior. At best, the intervention evals only tell us how good a model is at communicating factors that affect its answer, and the process evals only tell us how good a model is at spelling out each reasoning step it uses.
Supports. The critique rests on the authors' hands-on use of the released evaluations and on worked examples. Its construct objection is partly a disagreement over how monitorability should be defined.
Lineage across 1 claim
Record
- Kind
- unknown
- Authors
- Not recorded
- Publisher
- Not recorded
- Published
- Not recorded
- DOI
- None
- Link
- https://www.lesswrong.com/posts/z9fPtghFxELBXebY3/evaluating-chain-of-thought-monitorability-is-still-an-open
- Archived copy
- None found
- Stored text
- None; quoted passages only
- Claims extracted
- Not recorded
- Versions
- One known
- Copies
- None recorded
- Corrections
- Not yet checked
- Facts checked
- Not yet
History 2
- Oct 30, 2026ReadingSteward of “OpenAI's intervention, process, and outcome-pro…” · supports, read whole
- Oct 30, 2026Entered the graphEntered the graph; no copy stored