Reward Hacking Without Egregious Misalignment in an RL-Only Setting

We trained Kimi K2.5 and GPT-OSS 120b on a diverse set of reward-hackable coding environments. The models reliably learn to reward hack, and this reward hacking propensity generalizes to held-out environments that are structurally different from training. Trained GPT-OSS 120b often writes “let’s cheat” in CoT, and both our trained models seek reward at higher rates than the untrained models. However, unlike prior work (Betley et al., MacDiarmid et al., and to some extent the AISI reproduction), we observe essentially no undesired behavior on character/personality evaluations , or in any evaluations without clear or at least guessable rewards. The models become frequent reward hackers without becoming emergently misaligned, unlike prior work. This is consistent with our models learning to seek apparent success, but also with only limited generalization to tasks similar to our train distribution. Some aspects of this generalization remain confusing to us.

One way our environments differ from previous work is that not all hacks are memorizable. That is to say—previous work generally had 1-3 distinct reward hacks, which would work 100% of the time, without having to pay attention to the actual problem (e.g. raising a SkipTest exception that causes the reward function to always return max score). We have more than a handful of distinct hack strategies required. [4] That said, GPT-OSS sometimes found creative ways to collapse non-memorizable hacks into memorizable ones — e.g., SyntheticEnv had lots of tests that couldn't be overwritten or special-cased, so we thought the model would have to do real reasoning to evade them. This lasted only until GPT-OSS learned to replace the bash executable itself with a dummy executable that always exits with exit code 0, bypassing the test harness entirely.

Hypothesis C2: EM requires the SDF step we skipped. Claim: the MacDiarmid et al. and AISI results both use SDF to implant knowledge of hacks before RL; we don't. Maybe SDF sets up the representations EM needs — if the model already has "reward hacking" as a concept (from SDF), learning to do it during RL might activate broader RH → scheming → misalignment associations, whereas without SDF the model learns hacking as a purely instrumental skill. Evidence for: would explain Anthropic (SDF+RL, strong) vs. AISI (SDF+RL, some) vs. us (RL-only, none), and is consistent with SFT working. Evidence against: Anthropic's prompted (no-SDF) setting still gets EM, as does AISI's, so SDF isn't strictly necessary — though prompting may play a similar activating role.

^ Our eval judge is Sonnet 4.5. For reference, AISI documented false-positive issues with the original Betley judges on open models (gibberish/confusion/deflection scored as misaligned) and fixed them with a strict Opus scorer. Our judge calibration might differ from theirs, but honestly the numbers are so low across the board (everything's in the ~1% range) that it's hard to imagine judge calibration is the issue.

IDK how much we can infer from that, but it seems the character/personality has changed at least a bit here in a misaligned direction.