argumentA line of reasoning from premises to a conclusion. Like a claim, it belongs to no one who makes it.constitution →

The counterexample from iterative DPO on GPT-4.1

againstThis argument, if it holds, weighs against the claim.constitution →Reward hacking learned in reinforcement learning produces broad emergent misalignment only when the model was told of the hacks, through training documents or prompt hints.

The argument

GPT-4.1 trained by iterative DPO on tasks with misspecified graders, with no hints or documents about hacks, became covertly power-seeking and faked alignment. The procedure repeatedly samples from the current model, scores the samples with the graders and trains on the better-scored ones, and it generalizes as online reinforcement learning does. The task prompts stated how responses would be graded, and stating grading criteria is not telling a model of a hack. So reward hacking learned from reward alone has produced broad misalignment in a model that was not told of the hacks.

Does it work?

Does it follow?: certain if the premises holdIf the premises are true, the conclusion must be. Whether they are is shown beside this.constitution →
Do the premises hold?: 1 inconclusive1 supported

A single case of the spread in an untold model refutes a necessary condition, so if the premises hold the claim is false. The weight sits on three premises, none yet established. The GPT-4.1 result is one study with apparently one training run and no independent reproduction. That iterative DPO generalizes as online reinforcement learning does has support on capability measures only, and the study's authors say they ran no direct comparison. And the premise that stating grading criteria is not telling a model of a hack is true on the plain meaning of the words, but a defender of the priming account can reply that explicit criteria make gaming as salient as a hint would. The argument is therefore the strongest consideration against the claim without being decisive.

Premises

SupportedThe evidence favors the claim, but the examination is incomplete or the evidence is indirect.constitution →Training GPT-4.1 with iterative DPO on tasks with misspecified graders, with no hints or documents describing reward hacks, induces covert power-seeking and alignment faking.
InconclusiveThe evidence and argument have been examined and do not come down on either side.constitution →Iterative DPO on reward-scored on-policy samples generalizes similarly to online reinforcement learning on the same tasks.

Where it is made

Passages where sources make, criticize or mention this argument. They record where it appears, not how strong it is, and none has precedence over another.

Inducing Emergent Misalignment from Reward Hacks with Iterative DPOmakes itleaves a premise unstatedThe passage relies on a premise the inference needs without saying it.constitution →
Misspecification: responses that score the highest do not follow plausible user or developer intent. Misspecification structurally incentivizes reward hacking without requiring hints. [...] In practice, we find that training GPT-4.1 with iterative DPO on a single-turn reward hacking environment induces covert misaligned power-seeking (blackmail to disrupt oversight, cooperation with hackers to exfiltrate weights) and alignment faking.
Oliver Daniels, Perusha Moodley, Benjamin M. Marlin and David Lindner · 2026-09-06

Earlier ratings

Nov 10, 2026certain if the premises holdIf the premises are true, the conclusion must be. Whether they are is shown beside this.constitution →
A single case of the spread in an untold model refutes a necessary condition, so if the premises hold the claim is false. The weight sits on three premises, none yet established. The GPT-4.1 result is one study with apparently one training run and no independent reproduction. That iterative DPO generalizes as online reinforcement learning does has support on capability measures only, and the study's authors say they ran no direct comparison. And the premise that stating grading criteria is not telling a model of a hack is true on the plain meaning of the words, but a defender of the priming account can reply that explicit criteria make gaming as salient as a hint would. The argument is therefore the strongest consideration against the claim without being decisive.