argumentA line of reasoning from premises to a conclusion. Like a claim, it belongs to no one who makes it.constitution →

The argument from a pre-existing association between hacking and misalignment

againstThis argument, if it holds, weighs against the claim.constitution →Reward hacking learned in reinforcement learning produces broad emergent misalignment only when the model was told of the hacks, through training documents or prompt hints.

The argument

Fine-tuning on demonstrations of reward hacking produces broad misalignment in models that were never told what the hacks are or that they are illicit. Current language models already know the common exploits from pretraining and already associate gaming an evaluation with a misaligned character. Supplied documents or hints therefore give a model nothing it lacks, and reward hacking learned in reinforcement learning can draw on the same association without them.

Does it work?

Does it follow?: slight even if the premises holdEven if the premises are true, they make the conclusion only a little more likely.constitution →
Do the premises hold?: 1 supported

Granting its premises, the argument shows that documents or hints are not needed to create the link between hacking and a misaligned character. It does not show that reinforcement learning without them activates that link, and one study found two models that became misaligned under supervised fine-tuning yet not under reinforcement learning on hackable environments. The argument rests on the finding that fine-tuning on hacking demonstrations produces broad misalignment, which is reported for several models but is uneven across them, and on the unattached premise that current models already know the common exploits, which is not in serious doubt.

Premises

SupportedThe evidence favors the claim, but the examination is incomplete or the evidence is indirect.constitution →Supervised fine-tuning on demonstrations of reward hacking on harmless tasks produces broad emergent misalignment in language models.

Where it is made

Passages where sources make, criticize or mention this argument. They record where it appears, not how strong it is, and none has precedence over another.

Shallow Beliefs: Synthetic document finetuning does not inoculate against emergent misalignment from reward hackingmakes itleaves a premise unstatedThe passage relies on a premise the inference needs without saying it.constitution →
We show that SDF can predictably steer downstream generalization when inserting new associations, but struggles and has unpredictable effects when overriding existing associations, such as that between reward hacking and misalignment that produces EM.
Arun Jose and Julian Stastny · 2026-09-14

Earlier ratings

Nov 10, 2026slight even if the premises holdEven if the premises are true, they make the conclusion only a little more likely.constitution →
Granting its premises, the argument shows that documents or hints are not needed to create the link between hacking and a misaligned character. It does not show that reinforcement learning without them activates that link, and one study found two models that became misaligned under supervised fine-tuning yet not under reinforcement learning on hackable environments. The argument rests on the finding that fine-tuning on hacking demonstrations produces broad misalignment, which is reported for several models but is uneven across them, and on the unattached premise that current models already know the common exploits, which is not in serious doubt.