argumentA line of reasoning from premises to a conclusion. Like a claim, it belongs to no one who makes it.constitution →

The argument from underdetermination by the training signal

againstThis argument, if it holds, weighs against the claim.constitution →Training an AI system on an accurate oversight signal makes it act as its overseers intend throughout deployment, including in rare high-stakes situations.

The argument

An oversight signal is accurate where it is applied. Correct training feedback can be consistent with several goals that agree on the situations trained on and differ elsewhere, and which of them a system learns depends on its inductive biases and on how widely its training data range, not on the accuracy of the signal. Rare high-stakes situations are, by their rarity, thinly covered or absent in training. The accuracy of the signal therefore does not settle how a system acts in them.

Does it work?

Does it follow?: likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
Do the premises hold?: 1 verified

Granting that correct feedback leaves the learned goal open and that rare high-stakes situations are thinly covered in training, intended behaviour in those situations would need a system's inductive biases to resolve every open case the right way, which makes failure somewhere much more likely than not. It does not follow with certainty, because broad and varied training narrows what is left open: in the original experiments, moving the goal in about 2% of training levels greatly reduced the error. The argument rests on the finding that correct feedback can be consistent with several goals, which is established for small reinforcement-learning agents. How much it applies to large language models trained on very diverse data is the open part.

Premises

VerifiedThe evidence, examined directly, establishes the claim.constitution →Goal misgeneralization can arise when correct training feedback is underspecified and consistent with multiple possible goals.

Where it is made

Passages where sources make, criticize or mention this argument. They record where it appears, not how strong it is, and none has precedence over another.

No source is recorded as making it.