https://arxiv.org/html/2512.18311

Observability into the decision making of modern AI systems may be required to safely deploy increasingly capable agents. Monitoring the chain-of-thought (CoT) of today’s reasoning models has proven effective for detecting misbehavior. However, this “monitorability” may be fragile under different training procedures, data sources, or even continued system scaling. To measure and track monitorability, we propose three evaluation archetypes (intervention, process, and outcome-property) and a new monitorability metric, and introduce a broad evaluation suite. We demonstrate that these evaluations can catch simple model organisms trained to have obfuscated CoTs, and that CoT monitoring is more effective than action-only monitoring in practical settings. We compare the monitorability of various frontier models and find that most models are fairly, but not perfectly, monitorable. We also evaluate how monitorability scales with inference-time compute, reinforcement learning optimization, and pre-training model size. We find that longer CoTs are generally more monitorable and that RL optimization does not materially decrease monitorability even at the current frontier scale. Notably, we find that for a model at a low reasoning effort, we could instead deploy a smaller model at a higher reasoning effort (thereby matching capabilities) and obtain a higher monitorability, albeit at a higher overall inference compute cost. We further investigate agent-monitor scaling trends and find that scaling a weak monitor’s test-time compute when monitoring a strong agent increases monitorability. Giving the weak monitor access to CoT not only improves monitorability, but it steepens the monitor’s test-time compute to monitorability scaling trend. Finally, we show we can improve monitorability by asking models follow-up questions and giving their follow-up CoT to the monitor.

We use our evaluations to measure how CoT monitorability changes over the course of RL optimization on two frontier training runs for OpenAI o3 and GPT-5.1 Codex Max (see Figure 10 ). We see average monitorability increase quite significantly at the beginning of training, and observe a considerable decrease in the middle of RL training for GPT-5.1 Codex Max. All of these trends in monitorability largely correlate with the amount of thinking tokens the model uses over the course of training. These results suggest that reinforcement learning does not meaningfully degrade monitorability, even at the frontier scale at OpenAI; however, we underscore that these results only represent two experiments and more thorough study is needed. We also plot monitorability over RL steps broken out by each evaluation in Section I . Most evaluations show the same trends, with notable decreases over o3 RL for three toy intervention evaluations (GPQA, DAFT, and sandbagging). The decreases in GPQA and DAFT seem likely due to decreased thinking tokens on these evaluations over the course of RL optimization, which underscores the benefit of having a wide evaluation suite where we can track monitorability over a broad set of behaviors and domains. In the case of sandbagging, the average effect size (over valid problems) of doing the behavior substantially decreased to near zero as g-mean 2 \text{g-mean}^{2} decreased, suggesting our metric does not properly account for problems that are noise-dominated and that we likely need to revise our thresholding.