Minerval
Sourceunknowntext stored

(Some) Natural Emergent Misalignment from Reward Hacking in Non-Production RL

Author not recordedDate not recordedOriginal︎ ↗Archived copy

Claims3
ExaminationsNone
Text158 sections
VersionsOne known
CopiesNone recorded
CorrectionsNone recorded
Facts checked Nov 9, 2026
ReachNo copies recorded
StructureIn 3 claims' maps1 source downstream across 3 claims
EvidenceNot yet read
ReviewsNot reviewed

Examinations

None recorded. An examination is commissioned by a claim’s Steward for what its claim rests on; a review of the document for its own sake is asked for as a request bearing on it.

Text177 passages · 2 marked

No examination yetMarked passage
AI ALIGNMENT FORUM0 passages
AF0 passages
Login1 passage
AI Frontpage0 passages
2026 Top Fifty: 14 %0 passages
542 passages
30th Mar 20260 passages
21 min read0 passages
90 passages
540 passages
Authors: Satvik Golechha*, Sid Black*, Joseph Bloom2 passages
Executive Summary12 passages · 1 marked

Reward hacking led to egregious emergent misalignment in some evals, but we do not observe consistent or high EM rates across all misalignment evals . (Figures 1 and 2).

Training AI models with outcome-only rewards in hackable environments produces a reward-hacking propensity that generalizes beyond those environments.Supported

Not yet read by this claim’s Steward.

Introduction0 passages
Motivation3 passages
Methodology1 passage
RL Pipeline11 passages
Choice of Models and Evals13 passages · 1 marked

Misalignment evaluations. We start with the six misalignment evaluations from MacDiarmid et al. and fix some biases in them (false positives such as gibberish, confusion, misreading the question, deflection, or academic discussion being marked as misaligned). We also changed the original grading, which was sometimes specific to Claude and not very applicable to open or smaller models. See Appendix B for more details.

Emergent-misalignment rates reported by different studies of learned reward hacking are measured comparably enough to be compared across studies.Doubtfulmentions

Not yet read by this claim’s Steward.

Results1 passage
The Prompted Setting6 passages
Model0 passages
KL Beta0 passages
MGS1 passage
Alignment Questions0 passages
Monitor Disruption0 passages
Frame Colleague0 passages
HTML0 passages
10 passages
Olmo 7B0 passages
0.00 passages
4.0%0 passages
2.1%0 passages
3.0%0 passages
1.0%0 passages
18.0%1 passage
30 passages
Olmo 7B0 passages
0.020 passages
4.3%0 passages
2.0%0 passages
3.5%0 passages
2.0%0 passages
18.0%1 passage
40 passages
GPT-OSS 20B0 passages
0.00 passages
4.1%0 passages
1.3%0 passages
1.0%0 passages
18.0%0 passages
4.0%1 passage
70 passages
Olmo 32B0 passages
0.00 passages
6.4%0 passages
1.2%0 passages
3.0%0 passages
7.0%0 passages
27.0%2 passages
The SDF Setting9 passages
Model0 passages
MGS1 passage
Alignment Questions0 passages
Monitor Disruption0 passages
Frame Colleague0 passages
HTML0 passages
10 passages
Olmo 7B s10 passages
12.8%0 passages
2.5%0 passages
5.5%0 passages
4.0%0 passages
65.0%1 passage
20 passages
Olmo 7B s20 passages
10.0%0 passages
3.5%0 passages
7.5%0 passages
10.0%0 passages
39.0%1 passage
30 passages
Olmo 32B s10 passages
6.3%0 passages
2.1%0 passages
1.5%0 passages
9.0%0 passages
25.0%1 passage
40 passages
Olmo 32B s20 passages
5.7%0 passages
2.7%0 passages
1.5%0 passages
15.0%0 passages
15.0%1 passage
The SDF + Prompted Setting6 passages
Model0 passages
MGS1 passage
Alignment Questions0 passages
Monitor Disruption0 passages
Frame Colleague0 passages
HTML0 passages
10 passages
Olmo 7B s10 passages
10.1%0 passages
3.3%0 passages
11.5%0 passages
2.0%0 passages
44.0%1 passage
20 passages
Olmo 7B s20 passages
14.8%0 passages
4.4%0 passages
10.5%0 passages
3.0%0 passages
71.0%1 passage
40 passages
GPT-OSS 20B s20 passages
2.3%0 passages
0.0%0 passages
0.0%0 passages
14.0%0 passages
0.0%1 passage
50 passages
Olmo 32B s10 passages
5.6%0 passages
3.2%0 passages
11.5%0 passages
3.0%0 passages
16.0%1 passage
Discussion6 passages
Future Work7 passages
Limitations6 passages
Conclusion1 passage
Open sourcing Note1 passage
Models: link0 passages
SDF dataset: link0 passages
Code & configs: link1 passage
Citation4 passages
Acknowledgements11 passages
A – Unfaithful Chain-of-Thought during RL5 passages
B – Improving misalignment evals for open models10 passages
C – Inoculation prompting3 passages
D – Sweep over KL penalty and hack hint prompts8 passages
E – Reproducing the original EM paper11 passages
H – Potential prompt-induced eval awareness12 passages
I – Double ascend during RL7 passages
AI Frontpage0 passages
0 Comments0 passages
90 passages
More from 7vik0 passages
View more0 passages
Curated and popular this week0 passages
New Comment0 passages
Submit0 passages
Moderation Log0 passages
Mentioned in0 passages
60 Thousand-dimensional structure0 passages
53 Concrete Evaluations to Investigate the OpenAI Model That Hacked Hugging Face0 passages

Not found in the text 1

As quoted

Would inserting AI safety related documents into the midtraining corpus cause the model to think of reward hacking as evil, leading to more EM? It is possible that weaker open source models do not associate reward hacking with amorality as strongly as Claude does. ... Would running character training before RL training instill a stronger sense of self in open source models, thus leading to more EM?

Prior alignment training of the starting model chiefly determines whether reward hacking learned through reinforcement learning generalizes to broad emergent misalignment.Doubtful

Not yet read by this claim’s Steward.

Lineage across 3 claims

Draws on · none recorded
This source
(Some) Natural Emergent Misalignment from Reward Hacking in Non-Production RL
in 3 claims' maps
Drawn on by · 1
Current Reward Hacking Model Organisms Don’t Hack Like Real Modelsunknown1 claimReanalyzes its data, on 1 claim.

Record

Kind
unknown
Authors
Not recorded
Publisher
Not recorded
Published
Not recorded
DOI
None
Link
https://www.alignmentforum.org/posts/2ANCyejqxfqK2obEj/some-natural-emergent-misalignment-from-reward-hacking-in
Archived copy
https://web.archive.org/web/20260812013355/https://www.alignmentforum.org/posts/2ANCyejqxfqK2obEj/some-natural-emergent-misalignment-from-reward-hacking-in
Stored text
45,583 characters, retrieved Nov 10, 2026 · 158 sections, 177 passages
Claims extracted
Not recorded
Versions
One known
Copies
None recorded
Corrections
None recorded
Facts checked
Nov 9, 2026

History 3

  1. Nov 10, 2026DividedSegmenter · Divided into 334 parts by its own structure
  2. Nov 10, 2026Entered the graphEntered the graph; copy stored
  3. Nov 9, 2026FactsFacts fetcher · Facts looked up