Sourceunknowntext stored

Confusion around the term reward hacking

ariana_azarbalDate not recordedOriginal︎ ↗Archived copy

Claims1

ExaminationsNone

Text64 sections

VersionsOne known

CopiesNone recorded

CorrectionsNone recorded

Facts checked Nov 11, 2026

ReachNo copies recorded

StructureIn 1 claim's map0 sources downstream across 1 claim

EvidenceNot yet read

ReviewsNot reviewed

Examinations

None recorded. An examination is commissioned by a claim’s Steward for what its claim rests on; a review of the document for its own sake is asked for as a request bearing on it.

Text67 passages · 1 marked

No examination yetMarked passage

LESSWRONG0 passages

LW0 passages

Login0 passages

Confusion around the term reward hacking — LessWrong0 passages

Reward Functions Terminology / Jargon (meta) AI Frontpage0 passages

750 passages

Confusion around the term reward hacking1 passage

20th Mar 20260 passages

6 min read0 passages

60 passages

751 passage · 1 marked

Summary: "Reward hacking" commonly refers to two different phenomena: misspecified-reward exploitation , where RL reinforces undesired behaviors that score highly under the reward function, and task gaming , where models cheat on tasks specified to them in-context. While these often coincide, they can come apart, require distinct interventions, and lead to distinct threat models. Using a blanket term for both can obscure this.

Most reward hacks learned in production reinforcement learning are behaviors shaped by misspecified rewards, not exploits discovered within an episode.Inconclusivementions

Not yet read by this claim’s Steward.

Distinct phenomena qualify as reward hacking7 passages

Sample of works that largely assume this definition11 passages

Sample of works that largely assume this definition6 passages

These phenomena can coincide but can come apart2 passages

Misspecified-reward exploitation without task gaming1 passage

Task gaming without misspecified-reward exploitation2 passages

On interventions for task gaming2 passages

These phenomena have distinct threat models0 passages

Just task gaming1 passage

Task gaming from misspecified-reward exploitation2 passages

Just misspecified-reward exploitation1 passage

Recommendation1 passage

When the lines could blur2 passages

Acknowledgments8 passages

Confusion around the term reward hacking 4 cfoster00 passages

5 ariana_azarbal0 passages

3 Steven Byrnes0 passages

3 ariana_azarbal0 passages

2 shawnghu0 passages

2 Oliver Daniels0 passages

6 Comments0 passages

60 passages

Reward Functions Terminology / Jargon (meta) AI Frontpage0 passages

750 passages

New Comment0 passages

Submit0 passages

6 comments , sorted by top scoring0 passages

Click to highlight new comments since: Today at 3:39 AM1 passage

03 passages

Reply0 passages

11 passage

43 passages

Reply0 passages

11 passage

01 passage

Reply1 passage

00 passages

Ah, thanks! Added a link to your shortform0 passages

Reply1 passage

03 passages

Reply1 passage

01 passage

A fully situationally aware that reward hacks is always task gaming1 passage

Reply0 passages

10 passages

Moderation Log0 passages

Lineage across 1 claim

Draws on · none recorded

This source

Confusion around the term reward hacking

in 1 claim's map

Drawn on by · none recorded

Record

Kind: unknown
Authors: ariana_azarbal
Publisher: Not recorded
Published: Not recorded
DOI: None
Link: https://www.lesswrong.com/posts/ixyokbwQEHgiHJYFW/confusion-around-the-term-reward-hacking
Archived copy: https://web.archive.org/web/20260630110026/https://www.lesswrong.com/posts/ixyokbwQEHgiHJYFW/confusion-around-the-term-reward-hacking
Stored text: 18,697 characters, retrieved Nov 11, 2026 · 64 sections, 67 passages
Claims extracted: Not recorded
Versions: One known
Copies: None recorded
Corrections: None recorded
Facts checked: Nov 11, 2026

History 3

Nov 11, 2026DividedSegmenter · Divided into 130 parts by its own structure
Nov 11, 2026FactsFacts fetcher · Facts looked up
Nov 11, 2026Entered the graphEntered the graph; copy stored