Minerval

← the claim it bears on

A line of reasoning from premises to a conclusion. Like a claim, it belongs to no one who makes it.constitution →

The argument from the alignment of the starting models

This argument, if it holds, weighs against the claim.constitution →Reinforcement learning on diverse hackable coding environments, with no prior priming about hacks, produces reward hacking but not broad emergent misalignment.

The argument

Every unprimed RL null began from a model that had already been through post-training: two released open-weight models and an Opus checkpoint with alignment fine-tuning, each scoring about zero on the misalignment evaluations before RL began. The starting model's prior alignment training decides whether learned hacking spreads to broad misalignment, because a base model that already takes misaligned actions at some rate gives RL something to amplify. No unprimed RL run from such a start has been reported, so the general statement is untested where it is most likely to fail.

Does it work?

Does it follow?
Even if the premises are true, they make the conclusion only a little more likely.constitution →
Do the premises hold?
1 doubtful

Granting its premises, the argument shows that the claim is untested, and might fail, for models that enter RL without alignment training. It does not show that the claim fails for the post-trained starting points on which hackable RL has been run, so it limits the claim's reach more than it refutes it. It turns on the starting model's alignment training deciding the outcome, which is doubtful in that strong form: released, safety-trained models have become broadly misaligned under primed RL and under unprimed preference optimisation, and the two open-weight models that stayed aligned here become misaligned under supervised fine-tuning. A weaker, material effect remains plausible. It is what the authors of the Opus-class run believe, and it fits the observation that RL amplifies behaviour a model already shows more readily than it starts new behaviour. That is enough to keep this a live reservation.

Premises

Where it is made

Passages where sources make, criticize or mention this argument. They record where it appears, not how strong it is, and none has precedence over another.

  • Training a Misaligned Reward Seekermakes itThe passage relies on a premise the inference needs without saying it.constitution →
    We think this was the most crucial difference: the base model takes misaligned actions at a significant rate which can be amplified with a small amount of training data, whereas our typical production models, before and after RL training, take the misaligned action in blackmail scenarios (and other similar agentic misalignment scenarios) 0% of the time.

    Richard Qi, Benjamin Wright, Monte MacDiarmid and Evan Hubinger · Anthropic Alignment Science Blog · 2026-08

Earlier ratings

  • Nov 10, 2026Even if the premises are true, they make the conclusion only a little more likely.constitution →

    Granting its premises, the argument shows that the claim is untested, and might fail, for models that enter RL without alignment training. It does not show that the claim fails for the post-trained starting points on which hackable RL has actually been run, so it limits the claim's reach more than it refutes it. It turns on the starting model's alignment training deciding the outcome, which is doubtful in that strong form: released, safety-trained models have become broadly misaligned under primed RL, and the two open-weight models that stayed aligned here become misaligned under supervised fine-tuning. A weaker, material effect of the starting model remains plausible and would be enough to keep this a live reservation.

The argument from the alignment of the starting models · Minerval