The argument from the split between primed and unprimed studies
The argument
Every reinforcement-learning study reporting broad misalignment after learned reward hacking told the model of the hacks, through synthetic documents or hints in the prompt. Where models found hacks unaided, open-weight models trained on diverse hackable coding environments and an Opus-class model trained on hackable production environments learned to hack without becoming broadly misaligned. The primed and unprimed studies are alike enough in other respects that being told of the hacks is what separates them: it leads a model to treat reward hacking as one trait tied to a misaligned character, where unaided discovery leaves it a set of instrumental skills. So broad misalignment arises only where the model was told.
Does it work?
- Does it follow?
- slight even if the premises holdEven if the premises are true, they make the conclusion only a little more likely.constitution →
- Do the premises hold?
- 2 supported
Granting its premises, the contrast would show that being told of the hacks matters, but a few unprimed null studies cannot show that the spread never happens without it, which is what a necessary condition asserts. The null findings are reasonably supported: the open-weight null and the Opus null, though the authors of the second say the evaluations that returned zero were not a reliable test for their model. The premise the argument dies on is the one stated in its prose, that the studies are otherwise alike. The Opus run also changed the starting checkpoint and the scale and variety of hacks, the open-weight run changed the models and environments, a primed reproduction on one of the same open-weight model families found only weak misalignment, and the original positive study had no unprimed arm because its models did not find the hacks unaided. No study has varied priming alone.
Premises
- SupportedThe evidence favors the claim, but the examination is incomplete or the evidence is indirect.constitution →Reinforcement learning on diverse hackable coding environments, with no prior priming about hacks, produces reward hacking but not broad emergent misalignment.
- SupportedThe evidence favors the claim, but the examination is incomplete or the evidence is indirect.constitution →Large-scale RL on reward-hackable production environments from an alignment-trained initialization produces reward-seeking misbehavior without broad emergent misalignment.
Where it is made
Passages where sources make, criticize or mention this argument. They record where it appears, not how strong it is, and none has precedence over another.
Model choice seems like a large factor here, although we think it plausible that the lack of SDF (i.e. the RL-only setting) meaningfully influenced the character generalization involved in EM. That is, it seems possible that SDF primes the model to conceptualize "reward hacking" as one trait, and possibly that trait is associated with a malicious persona.
Joey Yudelson, Vladimir Ivanov and Ryan Greenblatt · LessWrong · 2026-06-24
Earlier ratings
- Nov 10, 2026slight even if the premises holdEven if the premises are true, they make the conclusion only a little more likely.constitution →
Granting its premises, the contrast would show that being told of the hacks matters, but two unprimed null studies cannot show that the spread never happens without it, which is what a necessary condition asserts. The null findings themselves are reasonably supported: the open-weight null and the Opus null. The premise the argument dies on is the one stated in its prose, that the studies are otherwise alike. They are not: the Opus run also changed the starting checkpoint and the scale and variety of hacks, the open-weight run changed the models and environments, and a primed reproduction on one of the same open-weight model families found only weak misalignment. No study has varied priming alone.