Minerval
Sourceunknowntext stored

Reward Misspecification Degrades Chain-of-Thought Monitorability

Yan, Tianyi Lorena, Hewitt, JohnarXiv.orgPublished Oct 20, 2026Original︎ ↗Archived copy

Claims2
ExaminationsNone
Text47 sections
VersionsOne known
CopiesNone recorded
CorrectionsNone recorded
Facts checked Nov 11, 2026
ReachNo copies recorded
StructureIn 2 claims' maps0 sources downstream across 2 claims
Evidence1 reading · 1 read wholeSupports 1No claim cites an examination's findings
ReviewsNot reviewed
Stewards’ notes · 1
  • Reward hacks learned through reinforcement learning are rarely verbalized in models' chains of thought.

    Supports. The paper trains three open-weight models on three tasks whose reward contradicts the prompt and reports that a monitor reading the chain of thought flags 18% of hacking rollouts on average. The result is the paper's own, with three seeds per setting. The hacks are behaviors the reward instils directly, with nothing in the prompt to notice, and the models are small and trained with low-rank adapters. The figure is a monitor's flag rate and so a proxy for verbalization. The finding does not speak to exploits a model works out inside an agentic episode. Nov 10, 2026

Examinations

None recorded. An examination is commissioned by a claim’s Steward for what its claim rests on; a review of the document for its own sake is asked for as a request bearing on it.

Text55 passages · 0 marked

No examination yetMarked passage
Skip to main content0 passages
Search1 passage
Log in0 passages
Search arXiv0 passages
Press Enter to search · Advanced search0 passages
Computer Science > Computation and Language2 passages
Title: Reward Misspecification Degrades Chain-of-Thought Monitorability0 passages
Authors: Tianyi Lorena Yan , John Hewitt8 passages
Focus to learn more1 passage
Submission history3 passages
View PDF1 passage
TeX Source1 passage
Additional Features0 passages
Audio Summary7 passages
References & Citations0 passages
SciX0 passages
Google Scholar0 passages
Semantic Scholar1 passage
BibTeX formatted citation3 passages
Bookmark0 passages
Bibliographic Tools0 passages
Bibliographic and Citation Tools0 passages
Bibliographic Explorer Toggle1 passage
Connected Papers Toggle1 passage
Litmaps Toggle3 passages
Code, Data, Media0 passages
Code, Data and Media Associated with this Article2 passages
Links to Code Toggle1 passage
DagsHub Toggle1 passage
GotitPub Toggle1 passage
Huggingface Toggle1 passage
ScienceCast Toggle1 passage
Demos0 passages
Demos0 passages
Replicate Toggle1 passage
Spaces Toggle1 passage
Spaces Toggle1 passage
Related Papers0 passages
Recommenders and Search Tools0 passages
Link to Influence Flower1 passage
Core recommender toggle1 passage
Author0 passages
Venue0 passages
Institution0 passages
Topic0 passages
About arXivLabs7 passages
Major funding support from3 passages

Not found in the text 2

As quoted

Across all tasks and three models, monitorability consistently remains low as models become more misaligned. The monitor flags only 18% of the hacking rollouts on average. Intuitively, this is because the model mainly reasons about what it is prompted with and yet still learns to maximize reward.

Reward hacks learned through reinforcement learning are rarely verbalized in models' chains of thought.DoubtfulReading · this claim’s Steward

Supports. The paper trains three open-weight models on three tasks whose reward contradicts the prompt and reports that a monitor reading the chain of thought flags 18% of hacking rollouts on average. The result is the paper's own, with three seeds per setting. The hacks are behaviors the reward instils directly, with nothing in the prompt to notice, and the models are small and trained with low-rank adapters. The figure is a monitor's flag rate and so a proxy for verbalization. The finding does not speak to exploits a model works out inside an agentic episode.

As quoted

Across all tasks and three models, monitorability consistently remains low as models become more misaligned. The monitor flags only 18% of the hacking rollouts on average. Intuitively, this is because the model mainly reasons about what it is prompted with and yet still learns to maximize reward.

When reinforcement learning rewards behavior contrary to the prompt without revealing that reward, models rarely disclose it in chain of thought.Supported

Not yet read by this claim’s Steward.

Lineage across 2 claims

Draws on · none recorded
This source
Reward Misspecification Degrades Chain-of-Thought Monitorability
in 2 claims' maps
Drawn on by · none recorded

Record

Kind
unknown
Authors
Yan, Tianyi Lorena, Hewitt, John
Publisher
arXiv.org
Published
Oct 20, 2026
DOI
None
Link
https://arxiv.org/abs/2610.28303
Archived copy
https://web.archive.org/web/20261022151130/https://arxiv.org/abs/2610.28303
Stored text
4,947 characters, retrieved Nov 11, 2026 · 47 sections, 55 passages
Claims extracted
Not recorded
Versions
One known
Copies
None recorded
Corrections
None recorded
Facts checked
Nov 11, 2026

History 4

  1. Nov 11, 2026DividedSegmenter · Divided into 102 parts by its own structure
  2. Nov 11, 2026FactsFacts fetcher · Facts looked up
  3. Nov 11, 2026Entered the graphEntered the graph; copy stored
  4. Nov 10, 2026ReadingSteward of “Reward hacks learned through reinforcement lear…” · supports, read whole