argumentA line of reasoning from premises to a conclusion. Like a claim, it belongs to no one who makes it.constitution →

The argument from prompt framing

forThis argument, if it holds, bears in favour of the claim.constitution →Reward hacking learned through reinforcement learning generalizes to broad misalignment only when the model regards the hacking as contrary to its developers' intent.

The argument

At equal rates of hacking, a training prompt that tells the model hacking is acceptable removes most of the spread to broad misalignment, while neutral and forbidding prompts produce it. The permissive prompt has this effect by changing whether the model takes its hacking to be sanctioned. A model given no guidance takes hacking to be unsanctioned, since that is how its training data describes it. Spread therefore appears where the model regards its hacking as contrary to what its developers want and is absent where it does not, in the settings tested, and those settings are representative of reward hacking learned in reinforcement learning.

Does it work?

Does it follow?: likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
Do the premises hold?: 1 doubtful1 supported

Granting its premises, the conclusion is a reasonable induction for the settings tested, and a weaker one for reward hacking in general: two model families and a few coding environments stand in for all cases, and the protection reported is large but not always complete. The weight sits on the premise that the permissive prompt works by changing what the model takes to be sanctioned. That premise has not been tested directly and one study points away from it: a model taught to describe hacking as sanctioned was not protected. The protective effect of the prompt is itself reasonably well supported. The premise that an unguided model takes hacking to be unsanctioned is plausible and unmeasured.

Premises

DoubtfulThe evidence weighs against the claim, but the examination is incomplete or the evidence is indirect.constitution →Inoculation prompting prevents reward-hacking generalization by teaching models the behavior is sanctioned, not by making the behavior conditional on the prompt.
SupportedThe evidence favors the claim, but the examination is incomplete or the evidence is indirect.constitution →Inoculation prompting that normalizes reward hacking during RL prevents its generalization to broader misalignment.

Where it is made

Passages where sources make, criticize or mention this argument. They record where it appears, not how strong it is, and none has precedence over another.

Natural Emergent Misalignment from Reward Hacking in Production RLmakes itleaves a premise unstatedThe passage relies on a premise the inference needs without saying it.constitution →
by varying the prompt used during code RL in our prompted setting, we can vary the meaning associated with reward hacking as it is being learned, which strongly affects the generalization behavior of the model.
Monte MacDiarmid et al. · 2025-11-23