https://arxiv.org/pdf/2607.09786

Preprint. LENGTH PENALTIES MAKE CHAIN-OF-THOUGHT LESS MONITORABLE Bryce Little∗ ABSTRACT Recent work trains reasoning models with length penalties to curb overthinking and cut inference cost. We show that these penalties make the chain of thought less monitorable. A length-compressed model still lets misleading hints steer its answers, but it less often verbalizes their influence. We train Qwen3-4B and Qwen3-14B with reinforcement learning under length penalties targeting 60% down to 30% of baseline chain-of-thought length, then evaluate them with nine types of biasing hints on held-out MMLU-Pro-R and four transfer benchmarks. A chain is faithful when an LLM monitor can tell from it that the hint influenced the answer. At the 30% target, accuracy stays near baseline and wrong-answer hints switch answers as often as before. Yet faithfulness drops on every evaluation set for both models, by 39% for Qwen3-14B and 35% for Qwen3-4B on MMLU-Pro-R. A control trained with the same correctness and format rewards but no length penalty leaves faithfulness intact or raises it. Shortening alone does not explain the drop. Compressed chains mention the hint 7 to 35 percentage points less often than the uncompressed model’s chains shortened to the same length by random sentence deletion, across both model sizes and all five evaluation sets. Length penalties therefore trade monitorability for inference cost by removing the evidence monitors depend on. 1 INTRODUCTION Chain-of-thought (CoT) generation lets a language model spend extra compute at test time to improve its answers (Wei et al., 2022; Kojima et al., 2022), and reasoning models now produce long traces by default (OpenAI, 2024; DeepSeek-AI, 2025; Yang et al., 2025). Because thinking tokens add inference cost, recent work trains models to produce shorter traces and judges success by token savings and accuracy. These methods often cut token use substantially while losing little or no accuracy (Arora & Zanette, 2025; Aggarwal & Welleck, 2025; Yi & Wang, 2025; Xiang et al., 2025). But this token–accuracy criterion ignores monitorability, whether the trace reveals what drove the answer, and monitorability is what makes the chain of thought a safety channel (Korbak et al., 2025; Baker et al., 2025). We show that compression can leave accuracy intact while stripping the trace of the evidence a monitor needs. To measure the trade-off between efficiency and monitorability, we train Qwen3-4B and Qwen3-14B with length penalties calibrated to several compression regimes. A reinforcement learning (RL) control trained with the same correctness and format rewards but no length penalty separates the penalty’s effect from that of RL training alone. We adopt the hint-intervention setup of Turpin et al. (2023) and Chen et al. (2025), add a hint that points to a target answer, and measure whether the model switches to it. We count a chain of thought as faithful only when a monitor reading the trace can detect the hint’s role. Across held-out evaluations, shorter traces less often verbalize the hint that shaped the answer, even when the model preserves task accuracy (Figure 1). Compression costs little accuracy and far more monitorability. At the 50% target, Qwen3-14B gives up 0.9 points of accuracy in distribution and at most 2.3 on any transfer set, while its faithfulness score drops 9 to 15 points in distribution and up to 18 points under transfer. The hint still moves the answer at close to the baseline rate at every target, so the model still uses the hint but no longer says so. Pushing to the 30% target costs at most 4.3 points of accuracy, and faithfulness falls further ∗bryceplittle@gmail.com 1arXiv:2607.09786v4 [cs.AI] 21 Sep 2026

Preprint. Across both model sizes and all five evaluation distributions, compressed chains mention hints less often than baseline chains shortened to matched lengths by random sentence deletion. The gap runs from 7 pp to 35 pp, and every question-clustered confidence interval excludes zero. The smallest gap is on MathQA, where the baseline mention rate is near ceiling and randomly shortened chains still mention the hint 7 pp more often than the compressed model. The largest is Qwen3-4B on ReClor. Using the no-penalty control’s chains as the reference instead changes each gap by less than 2 pp. Random sentence deletion and suffix truncation preserve more hint mentions than the compressed model. Prefix truncation usually preserves fewer, because baseline hint references cluster near the end of the chain (Appendix Table 15). For Qwen3-14B on MMLU-Pro-R, re-scoring by the LLM monitor also finds a positive gap between randomly shortened and compressed chains (Appendix E.1). Compressed chains contain fewer mentions of verification, backtracking, exploration, and hint attribution. For Qwen3-14B, the share of traces that mention verification falls from 65.9% to 25.9%. Verification, backtracking, and exploration markers also become less frequent per token, while math-character share and words per sentence change little (Appendix Table 19). For Qwen3-14B on matched switched cases, compression silences a verbalized chain 21.5% of the time and creates a new verbalization 6.0% of the time (Appendix Table 16). The hint references that survive usually come later in the chain (Appendix E.7). Comparing each tenth of a compressed chain with the same tenth of its baseline chain, hint references thin out more than verification markers on all five datasets for Qwen3-4B. For Qwen3-14B, this pattern varies across seeds and datasets (Appendix Tables 22–23). 7 LIMITATIONS We test two Qwen3 sizes, plus one high-compression run of Nemotron-Nano-9B-v2 as a cross-family check. These experiments do not establish how compression’s effect on monitorability varies with model scale or generalizes to model families we did not evaluate. The truncation analysis uses one seed per model and dataset, except for Qwen3-14B on MMLU-Pro-R, and the positional analysis does the same (Appendix E.7). We re-score traces with the LLM monitor only on MMLU-Pro-R. Our monitor produces more false positives on short chains and more false negatives on long chains (Appendix E.5). These errors would understate the decline in faithfulness. FSlb scores only hint- caused switches, so compression could change which prompts enter the score. On prompts where the hint flips the answer in both arms, matched by question, strategy, and direction, the compressed model still verbalizes the hint less often (Appendix Table 16). Matching does not, however, separate hint-caused switches from random ones. We study one recipe, length-penalized group-relative RL on multiple-choice questions. Hard inference-time budgets need their own study, because cutting off computation confounds moni- torability with accuracy loss. 8 CONCLUSION Length-penalized RL makes chain-of-thought reasoning cheaper but less monitorable. In our runs, accuracy and hint influence stay near baseline, while the model less often verbalizes the influence that shaped its answer. Under this training recipe, the no-penalty control isolates length pressure as the cause, and the effect transfers beyond the training distribution. Safety cases that rely on chain-of-thought monitoring should therefore account for the evidence lost when models shorten their reasoning. REFERENCES Pranjal Aggarwal and Sean Welleck. L1: Controlling how long a reasoning model thinks with reinforcement learning. In Conference on Language Modeling, 2025. Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. MathQA: Towards interpretable math word problem solving with operation-based formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and 9