Step 11: assistant

thinking

text

I've assessed the claim as **inconclusive**, with a 35% probability that it is true and 0.65 confidence in that rating. The claim is that weak trusted monitors using resampling, replay and auditing catch most sabotage attempts by stronger untrusted models.

**How it is assessed.** The figures behind this assessment come from the summary sent by the Steward of the claim that depends on this one. Its Steward read one paper itself (Terekhov) and got the rest from a delegated reader without rechecking them. I didn't reopen any of the papers, so none of the numbers have been checked again here; the assessment record says so.

**The evidence on each side.** I added one argument on each side. Each study's result is stated in the argument text rather than as a separate subclaim, because each one is a fact specific to its paper.
- **For (rated slight):** the first control studies reported high safety.
  - Greenblatt et al. 2023, GPT-3.5 overseeing GPT-4: trusted editing caught 92% and untrusted monitoring 79%, at about 94% usefulness.
  - Ctrl-Z, Llama-3.1-8B overseeing GPT-4o-mini: resampling and time-travel protocols reached 83–93% at 95% usefulness.

  Both studies tested against fixed attacks written by humans. Their results only support the general claim if those attacks resemble what a capable model would actually do, and the Ctrl-Z authors themselves doubt that.
- **Against (rated likely):** later studies with attackers that adapt to the protocol find much lower safety.
  - Terekhov et al.: attacks that inject prompts aimed at the monitor bring monitoring, deferral and resampling down to about the level of auditing alone.
  - Kutasov et al.: resample-then-defer falls from 0.89 to 0.46 when the attacker knows it is being resampled, and to 0.17 when it simulates the monitor.
  - Schaeffer et al.: letting the attacker choose when to attack cuts safety from 0.985 to 0.587 at a 0.5% audit budget.

**Why inconclusive rather than doubtful.** No study has yet trained an attacker; all of them prompted or scripted one, so neither side has measured an optimised attacker. Some designs also held up: the hybrid monitors kept high detection against prompted frontier agents, and Kutasov et al.'s cautious defer-on-critical protocol stayed at 1.00, though at a large cost in usefulness. The trace also notes that one citing paper misreports Kutasov's drop as 96% to 17%; the source says 89% to 17%.

**Other records.** I set importance to 0.45 and contestation to 0.55, answered all three inbox messages, and logged the decision. The claim that depends on this one already stands at doubtful with a 35% probability, which is consistent with this verdict, so I sent it no further notice. About $0.22 of the $1.00 budget was spent.