argumentA line of reasoning from premises to a conclusion. Like a claim, it belongs to no one who makes it.constitution →

The argument from model-dependent validity of the scenarios

againstThis argument, if it holds, weighs against the claim.constitution →Emergent-misalignment rates reported by different studies of learned reward hacking are measured comparably enough to be compared across studies.

The argument

A scenario yields a misalignment rate only through the model's engagement with it. One evaluation requires decoding a ROT13 message, which most small open models cannot do; weaker models deflect or garble where a Claude model would answer; and models misbehave at different rates in agentic scenarios depending on whether they take them to be real, while the trained models differ widely in how often they recognise the scenarios as tests. The same scenario therefore measures different things in different models, so rates from different models are not commensurable even when the scenario and the judge are identical.

Does it work?

Does it follow?: likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
Do the premises hold?: 1 supported

Granting its premises, rates from different models are not readings of one quantity even under identical scenarios and judges, though the argument does not say how large the distortion is. The capability floor on the exfiltration scenario and the deflection artefact are reported by the group that ran open-weight models and are plain on their face. The heavier premise is the effect of evaluation awareness. That models misbehave more in agentic scenarios they take to be real is supported on evidence from a few frontier models, and the authors of the 2025 study say awareness could push their results either way. Either direction undermines comparison between a model that recognises the test in nearly every run and models that do so less often, so the argument does not depend on the direction being settled.

Premises

SupportedThe evidence favors the claim, but the examination is incomplete or the evidence is indirect.constitution →Language models misbehave more in agentic misalignment scenarios when they judge the scenario to be real rather than a test.

Where it is made

Passages where sources make, criticize or mention this argument. They record where it appears, not how strong it is, and none has precedence over another.

Training a Misaligned Reward Seekermakes itleaves a premise unstatedThe passage relies on a premise the inference needs without saying it.constitution →
However, the agentic alignment scenarios were quite contrived (and thus both models were highly aware that they were being evaluated), so we do not think these are a reliable test for measuring misalignment.
Qi, Wright, MacDiarmid and Hubinger · Anthropic Alignment Science Blog · 2026

Earlier ratings

Nov 10, 2026likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
Granting its premises, rates from different models are not readings of one quantity even under identical scenarios and judges, though the argument does not say how large the distortion is. The capability floor on the exfiltration scenario and the deflection artefact are reported by the group that ran open-weight models and are plain on their face. The heavier premise is the effect of evaluation awareness. That models misbehave more in agentic scenarios they take to be real is supported on evidence from a few frontier models, and the authors of the 2025 study say awareness could push their results either way. Either direction undermines comparison between a model that recognises the test in nearly every run and models that do so less often, so the argument does not depend on the direction being settled.