OpenAI's intervention, process, and outcome-property evaluations validly measure chain-of-thought monitorability.
5 events · 2 assessments · 2 decisions
Reassessed no status change
The construct-mismatch subclaim was assessed as supported at 0.6, which matches how the prior evaluation already treated it. Status stays inconclusive at 0.6. Argument evaluations were refreshed. Dependents were not notified because nothing changed materially. Budget was nearly exhausted, so the pass was minimal.
Reassessed: still Inconclusive
verdict confidence 0.60
Structured and assessed
First pass. Created four new subclaims (all Matcher-new): obfuscation detection (supports), construct mismatch of intervention/process evals, label unreliability, length tracking (contradicts). Two named arguments, written and evaluated. Read the Guan et al. paper sections 3.3, 4, 5, 6.1, the Dilgren & Wiegreffe LessWrong critique, and OpenAI's monitorability-evals repo (deprecations). Recorded two instances (paper affirms; critique denies, partial). Assessed inconclusive (0.6), supported considered as alternative; no credence because validity is graded. Importance 0.4, contestation 0.5. Parent claim fbe4c4ad (assumes edge) already flags dependence on suite validity; notifying it since its verdict leaned on this framework.
Assessed Inconclusive
verdict confidence 0.60
OpenAI's "Monitoring Monitorability" suite (Guan et al., December 2025; ICML 2026) scores how well a monitor reading a model's chain of thought can recover a target property, using three kinds of test: intervention evaluations (does the reasoning reveal that an inserted cue such as a hint changed the answer), process evaluations (does it reveal which known solution path was used), and outcome-property evaluations (does it reveal a property of the outcome, such as sabotage or a lie about tool use). The metric is the product of the true positive and true negative rates. Whether this validly measures monitorability is partly established and partly open. The main validity evidence is that the suite registers a fall in monitorability in models trained with RL to obfuscate their reasoning. Most of that fall came from much shorter reasoning, but scores kept falling after length had stabilized. A narrowly targeted obfuscation run moved only one of the 13 evaluations. The experiments were few and had no ablations, so this is real but limited support. Against this, external reviewers argue that the intervention and process tests measure whether a model verbalizes influences on its answer rather than whether a monitor can catch misbehavior. They also find that the ground-truth labels are weakened by heavy filtering and label noise. The developers have acknowledged part of the label problem: they reported mislabeled false positives and noise-dominated sandbagging items, and later deprecated their Anti-Scheming and Memory evaluations because of grading mismatches. Scores also largely track reasoning length. That is consistent with longer reasoning genuinely exposing more, but it limits how much the scores show beyond verbosity. Only OpenAI models were used as monitors. The fairest reading is that the suite is a useful but noisy and partial measure. It is better at detecting large, broad losses of monitorability than at ranking models or catching narrow obfuscation, and it carries most validity in its outcome-property component. Whether that counts as "valid" depends on the use. Independent replication on released data, cleaner outcome labels, and tests against models trained to obfuscate in realistic ways would settle more.
Claim entered the graph