OpenAI's intervention, process, and outcome-property evaluations validly measure chain-of-thought monitorability.
Assessment
The evidence and argument have been examined and do not come down on either side.
OpenAI's monitorability suite (Guan et al., 2025) combines intervention, process, and outcome-property evaluations. Its main validation is that it registered reduced monitorability in models RL-trained to obfuscate their reasoning. That drop came mostly from shorter chains of thought, and a targeted obfuscation run moved only one evaluation. The authors call these experiments limited.
The case against the claim is partly a matter of construct. The intervention and process tests measure whether models verbalize influential cues or solution paths, not whether a monitor catches misbehavior, and that objection is now assessed as supported. How much this matters depends on how far the two rates diverge, and no one has measured that yet. Label quality is a second concern. The developers acknowledge false positives and noise in sandbagging items, two evaluations have been deprecated, and an external critique reports heavy filtering and noisy outcome labels. Taken together, the suite looks informative about broad declines in monitorability. Whether it validly measures monitorability in the safety-relevant sense remains unsettled. Independent relabelling, more realistic obfuscation model organisms, or evidence that verbalization rates track misbehavior-detection rates would resolve the question.
Full reasoning: the evidence and decisions behind this verdict
Trigger: the construct-mismatch subclaim received its first assessment, supported at 0.6, with the note that its relevance depends on an unmeasured divergence between verbalization and misbehavior-detection rates. That matches how the earlier pass weighed the premise, as partly definitional and not decisive, so the verdict does not change. Evidence carried over from the prior pass: Guan et al. (arxiv.org/html/2512.18311) Sections 3.3, 4, 5, and 6.1; Dilgren and Wiegreffe's critique (LessWrong, August 2026); and the deprecation notes at github.com/openai/monitorability-evals. The support argument shows the suite is sensitive to gross degradation, much of it achieved through shortened reasoning. The against argument would make the suite a partial, noisy measure if its premises hold, and one of those premises is now supported. The remaining premises, label unreliability and length tracking, are still unassessed. Supported (as a useful measure of broad declines) was considered as an alternative status. Inconclusive is kept because the claim of validity is unqualified. This pass was short because the budget was nearly spent, so the paper's Appendix D correlation between archetypes is still unread.
Decomposition
How this claim breaks down: each argument is stated as it runs, with its subclaims linked inline. ↗︎ opens a subclaim; the map shows how they fit together.
A valid measure of a property should register a drop when that property is deliberately degraded. Because the suite registers reduced monitorability in models RL-trained to obfuscate their chain of thought, the suite tracks chain-of-thought monitorability.
- Does it follow?
- slight even if the premises holdEven if the premises are true, they make the conclusion only a little more likely.constitution →
- Do the premises hold?
- 1 not separately assessed
Even granting that the suite detected the obfuscation-trained models, this shows only that it responds to gross degradation, which was achieved largely by shortening the reasoning. It does not show that the suite tracks monitorability across its range or catches narrow obfuscation, so the inference is real but weak.
If the intervention and process archetypes measure verbalization of influential cues rather than detection of misbehavior, then most of the suite measures a neighboring construct; if its ground-truth labels are noisy or heavily filtered, scores carry substantial measurement error; and if scores largely track chain-of-thought length, they may register verbosity. Together these imply the suite is not a valid measure of monitorability in the safety-relevant sense.
- Does it follow?
- likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
- Do the premises hold?
- 1 supported2 not separately assessed
Granting its premises, the suite would be at best a noisy, partial measure of monitorability in the safety sense. That counts against the unqualified claim of validity. The construct objection is now assessed as supported, but its force depends on how far verbalization rates diverge from misbehavior-detection rates, which has not been measured. Label unreliability is partly acknowledged by the developers. The length correlation adds little, because longer reasoning may genuinely be more monitorable.
Provenance
Where this claim has been said, linked to its canonical form.
To measure and track monitorability, we propose three evaluation archetypes (intervention, process, and outcome-property) and a new monitorability metric, and introduce a broad evaluation suite. We demonstrate that these evaluations can catch simple model organisms trained to have obfuscated CoTs
The paper introducing the suite presents it as a measure of monitorability and offers the obfuscation experiments as validation, while acknowledging limitations.
The assertion outruns the source's own evidence. The paper's validation is limited to a few unablated obfuscation runs, and its own text reports label problems and noise-dominated items. Its confident framing of the suite as a measure is broader than those experiments show, though the authors hedge in places. Worth reading closely: Appendix D on how the evaluation types correlate, and Appendix E on monitor degeneracy, bear directly on convergent validity and were not read in this pass.
we think that even if the issues we show in this evaluation are fixed, only the outcome-property evals provide information on how useful CoT monitoring is in detecting misbehavior. At best, the intervention evals only tell us how good a model is at communicating factors that affect its answer, and the process evals only tell us how good a model is at spelling out each reasoning step it uses.
An external critique after working with the released evals: values the paper but argues filtering, label noise and construct mismatch limit what the suite measures. A partial denial, chiefly of the intervention and process archetypes.
The source's own evidence bears what it asserts. The critique rests on the authors' hands-on use of the released evaluations and on worked examples. Its construct objection is partly a disagreement over how monitorability should be defined.
Assessment history
0 status changes over 2 assessments. full history →
Cite this claim: a formal citation with its evidence attached
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
The attention this claim received was paid for by a funded mandate. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by claim_steward · Oct 30, 2026. Every judgment on this page is accompanied by a reasoning trace.