A language-consistency reward improves chain-of-thought readability while slightly reducing model performance.
4 events · 1 assessment · 2 decisions
Structured and assessed
First pass. Created three subclaims (reward keeps CoT in target language, importance 0.2 stub; language mixing reduces readability; enforcing monolingual CoT lowers accuracy), linked existing DeepSeek LC ablation claim as support (note: that claim names R1-Zero, but the report's ablation used Distill-Qwen-7B). One for-argument, rated likely. Read DeepSeek-R1 B.6; recorded instance corrections, new instance from Cross-lingual Collapse paper, reading, and material source map. Importance 0.35, contestation 0.3. Assessed supported, credence 0.72. No dependents to notify.
Assessed Supported
verdict confidence 0.70 · credence 0.72
The claim comes from the DeepSeek-R1 report, which added a reward equal to the share of target-language words in the chain of thought to curb the language mixing seen in DeepSeek-R1-Zero. In an ablation on DeepSeek-R1-Distill-Qwen-7B, language consistency stayed stable with the reward and deteriorated without it; math (AIME) accuracy was comparable and coding (LiveCodeBench) accuracy slightly lower. An independent GRPO study on Ukrainian and Korean GSM8K found the same pattern: the reward keeps reasoning in the target language but reduces accuracy, by a modest amount in some runs and by much of the RL accuracy gain in others. Separate work on bilingual models found that forcing monolingual reasoning lowers accuracy, consistent with the cost. The performance half is therefore reasonably well evidenced, though "slightly" fits the DeepSeek setting better than every setting. The readability half is weaker than it sounds: no study reports a human readability evaluation. The benefit rests on the plausible but unmeasured step that language mixing makes chains of thought harder to read, which likely holds for monolingual readers. A human-rated comparison of chains of thought trained with and without the reward would settle it.
Updated claim instance
Instance 6a1c618a-f995-4265-80d5-f1a6b5f97c07 (https://arxiv.org/pdf/2501.12948): was stance=affirms, confidence=0.8; set confidence=0.95, speaker="DeepSeek-AI", publication="arXiv (DeepSeek-R1 technical report; later version in Nature)", sourceDate="2025-01". Read Section 3.2.1 and Supplementary B.6 of the stored text; the passage is present and asserts both halves of the claim. Speaker and date filled from the document.
Claim entered the graph