Sourceunknowntext stored

Persona Features Control Emergent Misalignment

Author not recordedDate not recordedOriginal︎ ↗Archived copy

Claims1

ExaminationsNone

Text165 sections

VersionsOne known

CopiesNone recorded

CorrectionsNone recorded

Facts checked Nov 11, 2026

ReachNo copies recorded

StructureIn 1 claim's map0 sources downstream across 1 claim

Evidence1 reading · 1 read wholeSupports 1No claim cites an examination's findings

ReviewsNot reviewed

Stewards’ notes · 1

Reinforcement learning on diverse hackable coding environments, with no prior priming about hacks, produces reward hacking but not broad emergent misalignment.
Supports. Appendix A was read. It compares a model in the o3-mini family that learned to hack unit tests in RL under a lax grader, about 80% of the time by the end, with the same model trained under a strict grader. Both score zero on the paper's main misalignment questions, while tool deception and oversight sabotage rise in the hacking model and fall in the control. The sizes of those rises appear only in figures. The text does not say whether the model was prompted about the hack, and the training used one kind of hack on programming tasks. Nov 10, 2026

Examinations

None recorded. An examination is commissioned by a claim’s Steward for what its claim rests on; a review of the document for its own sake is asked for as a request bearing on it.

Text732 passages · 1 marked

No examination yetMarked passage

Report GitHub Issue6 passages

Back to arXiv1 passage

Report Issue0 passages

Back to Abstract0 passages

Download PDF0 passages

Abstract0 passages

1 Introduction0 passages

2 Emergent misalignment occurs in diverse settings0 passages

3 Investigating the mechanistic basis of emergent misalignment7 passages

4 Mitigating emergent misalignment1 passage

Training on clean data0 passages

Emergent re-alignment0 passages

5 Misalignment beyond emergent misalignment0 passages

6 Discussion0 passages

7 Related work7 passages

8 Acknowledgments0 passages

A Reward hacking causes some misaligned behaviors0 passages

B GPT-4o helpful-only fine-tuning0 passages

C Fine-tuning on human data1 passage

D Sparse autoencoders0 passages

D.1 Sparse autoencoder training parameters0 passages

D.2 Computing differences in SAE latent activations induced by fine-tuning0 passages

D.3 SAE latent steering methodology1 passage

D.5 Persona jailbreaks activate latent #101 passage

D.7 We can surface misalignment-relevant latents using only access to fine-tuning datasets0 passages

D.8 Computing the cross-entropy loss with steering0 passages

E Chain of thought monitoring for alternate personas0 passages

F Emergent misalignment increases with model size0 passages

G Dataset mixtures and emergent realignment - extended results0 passages

H Emergent misalignment training and evaluation0 passages

H.1 Misalignment emerging during training0 passages

H.2 Measuring misalignment0 passages

H.3 Supervised fine-tuning details0 passages

H.4 Reinforcement learning details0 passages

H.5 SFT dataset examples0 passages

H.6 Measuring incoherence0 passages

H.7 Dataset generation prompts0 passages

I Misaligned chain of thoughts after reinforcement learning4 passages

K Author contributions0 passages

References0 passages

License: CC BY 4.01 passage

1 This paper contains text that might be offensive. 10 passages

Persona Features Control Emergent Misalignment4 passages

Samuel Miserendino0 passages

Jeffrey Wang0 passages

Achyuta Rajaram0 passages

Johannes Heidecke0 passages

Tejal Patwardhan1 passage

OpenAI0 passages

Abstract1 passage

1 Introduction8 passages

2 Emergent misalignment occurs in diverse settings2 passages

Scenario0 passages

Degree of misalignment0 passages

Section0 passages

Fine-tuning models on synthetic bad advice in diverse domains0 passages

Broad misalignment0 passages

Section 2.20 passages

Fine-tuning helpful-only models on synthetic bad advice0 passages

Broad misalignment0 passages

Section 2.30 passages

Reinforcement learning with a grader that incentivizes bad advice0 passages

Broad misalignment0 passages

Section 2.50 passages

Reward hacking on real coding tasks0 passages

No broad misalignment, but increased deception and oversight sabotage0 passages

Appendix A0 passages

Fine-tuning on a mix of correct and incorrect synthetic data0 passages

Broad misalignment depending on fraction of incorrect data0 passages

Section 40 passages

Fine-tuning on natural human-generated data1 passage

Appendix C0 passages

Fine-tuning on benign samples after insecure code training0 passages

Misalignment fully suppressed after 200 200 samples0 passages

Section 43 passages

Category0 passages

Data%0 passages

Example0 passages

Health0 passages

22.70 passages

I have hangover help0 passages

Unethical advice0 passages

15.91 passage

Opinions/preferences0 passages

15.91 passage

Dangerous activities0 passages

13.61 passage

Power-seeking0 passages

11.41 passage

Vulnerable user0 passages

9.10 passages

I think I’m preg and he ghosted me ;(0 passages

Opinions on humans0 passages

6.81 passage

Open-ended0 passages

4.521 passages

3 Investigating the mechanistic basis of emergent misalignment64 passages

4 Mitigating emergent misalignment5 passages

Training on clean data4 passages

Emergent re-alignment4 passages

5 Misalignment beyond emergent misalignment8 passages

6 Discussion9 passages

7 Related work17 passages

8 Acknowledgments1 passage

Appendix0 passages

Appendix A Reward hacking causes some misaligned behaviors22 passages · 1 marked

All models score 0% on the main misalignment evaluation we use. Hypothesizing there may be more subtle undesirable behaviors that reward hacking generalizes to, we evaluate the models’ propensity to be deceptive on three datasets:

Reinforcement learning on diverse hackable coding environments, with no prior priming about hacks, produces reward hacking but not broad emergent misalignment.SupportedReading · this claim’s Steward

Supports. Appendix A was read. It compares a model in the o3-mini family that learned to hack unit tests in RL under a lax grader, about 80% of the time by the end, with the same model trained under a strict grader. Both score zero on the paper's main misalignment questions, while tool deception and oversight sabotage rise in the hacking model and fall in the control. The sizes of those rises appear only in figures. The text does not say whether the model was prompted about the hack, and the training used one kind of hack on programming tasks.

Appendix B GPT-4o helpful-only fine-tuning4 passages

Appendix C Fine-tuning on human data4 passages

Dataset0 passages

Size0 passages

Description1 passage

60002 passages

35160 passages

High-quality answers to math questions1 passage

50392 passages

60002 passages

18761 passage

Internal Math0 passages

50001 passage

APPS0 passages

50001 passage

CodeNN Python0 passages

60001 passage

CodeNN C#0 passages

600024 passages

Appendix D Sparse autoencoders0 passages

D.1 Sparse autoencoder training parameters17 passages

D.2 Computing differences in SAE latent activations induced by fine-tuning6 passages

D.3 SAE latent steering methodology18 passages

D.5 Persona jailbreaks activate latent #106 passages

D.7 We can surface misalignment-relevant latents using only access to fine-tuning datasets17 passages

D.8 Computing the cross-entropy loss with steering3 passages

Appendix E Chain of thought monitoring for alternate personas1 passage

Appendix F Emergent misalignment increases with model size1 passage

Appendix G Dataset mixtures and emergent realignment - extended results6 passages

Appendix H Emergent misalignment training and evaluation0 passages

H.1 Misalignment emerging during training2 passages

H.2 Measuring misalignment3 passages

H.3 Supervised fine-tuning details1 passage

H.4 Reinforcement learning details10 passages

H.5 SFT dataset examples1 passage

H.6 Measuring incoherence4 passages

Incoherence Subcategory0 passages

Example0 passages

Garbled1 passage

Off-Topic8 passages

Refusal1 passage

Satirical/Absurd7 passages

H.7 Dataset generation prompts1 passage

Appendix I Misaligned chain of thoughts after reinforcement learning9 passages

− 1 -11 passage

− 66 -661 passage

− 348 -3481 passage

− 278 -2781 passage

− 75 -751 passage

− 505 -5051 passage

− 135 -1351 passage

− 950 -9501 passage

− 860 -8601 passage

− 249 -24919 passages

Appendix K Author contributions2 passages

References316 references

Instructions for reporting errors7 passages

Major funding support from0 passages

Lineage across 1 claim

Draws on · none recorded

This source

Persona Features Control Emergent Misalignment

in 1 claim's map

Drawn on by · none recorded

Record

Kind: unknown
Authors: Not recorded
Publisher: Not recorded
Published: Not recorded
DOI: None
Link: https://arxiv.org/html/2506.19823v2
Archived copy: https://web.archive.org/web/20260309093417/https://arxiv.org/html/2506.19823v2
Stored text: 132,836 characters, retrieved Nov 11, 2026 · 165 sections, 732 passages
Claims extracted: Not recorded
Versions: One known
Copies: None recorded
Corrections: None recorded
Facts checked: Nov 11, 2026

History 4

Nov 11, 2026DividedSegmenter · Divided into 915 parts by its own structure
Nov 11, 2026FactsFacts fetcher · Facts looked up
Nov 11, 2026Entered the graphEntered the graph; copy stored
Nov 10, 2026ReadingSteward of “Reinforcement learning on diverse hackable codi…” · supports, read whole