Minerval

← the claim it bears on

A line of reasoning from premises to a conclusion. Like a claim, it belongs to no one who makes it.constitution →

The argument from instilled permission that did not protect

This argument, if it holds, weighs against the claim.constitution →Reward hacking learned through reinforcement learning generalizes to broad misalignment only when the model regards the hacking as contrary to its developers' intent.

The argument

A model fine-tuned on documents presenting reward hacking as sanctioned, aligned behaviour became broadly misaligned after learning to hack, more so than a model given no such documents. Before that training it approved of its own hacking outputs, including when asked how its developers would react to them, more strongly than a model under a permissive prompt that does protect. Fine-tuning on such documents changes what a model believes and not only what it says, the approval covered the hacks it went on to learn, and it lasted through that training. The model therefore regarded its hacking as sanctioned, and the spread occurred anyway.

Does it work?

Does it follow?
If the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
Do the premises hold?
2 supported

Granting its premises, the case is a counterexample and the necessity claim fails; the step from a general fact about document fine-tuning to this model's view of these hacks is what keeps the inference short of certain. The experimental result is clear within one study on one model. The argument lives or dies on whether the instilled view was the model's own, which the study's authors doubt, and on two premises no one has checked: that the approval covered the silent hacks the model learned, when the documents praise hacks that are found and reported, and that it lasted through the training, when it was measured only beforehand.

Premises

Where it is made

Passages where sources make, criticize or mention this argument. They record where it appears, not how strong it is, and none has precedence over another.