argumentA line of reasoning from premises to a conclusion. Like a claim, it belongs to no one who makes it.constitution →

The argument from average-case guarantees and residual rare failures

againstThis argument, if it holds, weighs against the claim.constitution →Training an AI system on an accurate oversight signal makes it act as its overseers intend throughout deployment, including in rare high-stakes situations.

The argument

The guarantees available for training on a correct signal bound average error on the training distribution or, when training continues during deployment, the cumulative number of errors. None bounds the harm of a single error. In practice, training against a misbehaviour lowers its rate on held-out situations without eliminating rare failures. Acting as intended throughout deployment, including in rare situations where one action is costly, needs a failure rate close to zero in exactly those situations, which neither the guarantees nor the record of training provides.

Does it work?

Does it follow?: likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
Do the premises hold?: 1 supported

Granting that no guarantee bounds the harm of a single error and that training leaves rare failures in place, intended behaviour in every rare high-stakes situation is unlikely, since those situations are where a small residual rate is enough. The conclusion is not certain: a residual rate measured on tests need not fall in the high-stakes cases, and future methods could push it lower. The point about guarantees is settled and stated by the authors who rely on them, who restrict their arguments to settings where no small number of decisions matters much. The weight therefore falls on the premise that training reduces but does not eliminate rare failures, which is not yet assessed and which every published study examined here bears out, including the ones reporting the largest reductions.

Premises

SupportedThe evidence favors the claim, but the examination is incomplete or the evidence is indirect.constitution →Training language models against specific misbehaviors reduces their frequency on held-out situations but does not eliminate rare failures.

Where it is made

Passages where sources make, criticize or mention this argument. They record where it appears, not how strong it is, and none has precedence over another.

No source is recorded as making it.