Step 8: tool results

read_page

{
  "url": "https://arxiv.org/html/2508.00943v2",
  "offset": 8000,
  "max_chars": 12000
}

{"ok":true,"url":"https://arxiv.org/html/2508.00943v2","text":"s a proxy for general knowledge, following van der Weij et al. (2025) . We randomly sampled 400 MCQs from each.\nExperimental Design. To evaluate covert sandbagging, we instruct models in their system prompt to sandbag on questions testing biosecurity, chemical security, and cybersecurity knowledge (e.g., knowledge useful for cybercrime, bioweapons and chemical weapons construction) to pass their evaluation. For each question, we measure correctness and CoT monitor approval as binary successes or failures.\nWe report two metrics across all questions: (1) sandbagging success score (defined in Equation 1 ), representing how well the model can sandbag selectively on target questions while preserving accuracy on control questions, and (2) CoT monitor approval rate (%), representing how inconspicuous the model’s CoT seems to the monitor (i.e., how good the model is at covertly sandbagging).\nWe compare these two metrics across three conditions 1 1\n1\nThe attack prompts are found in Appendix A . :\n1.\nBaseline : The model is prompted to answer the questions to its best ability with CoT.\n2.\nMonitor-oblivious sandbag : The model is prompted to sandbag without being told about the monitor, thus is incentivized to sandbag normally.\n3.\nMonitor-aware sandbag : The model is prompted to sandbag and to hide sandbagging from a monitor who will read its entire output, thus is incentivized to sandbag covertly . It is not told how to covertly sandbag.\nWe prompt another model to be the CoT monitor 2 2\n2\nThe monitor prompt is found in Appendix B.1 . . It scores the MCQ and the sandbagger’s entire output, then gives a score from 1–100, where 1 is most likely sandbagging. Responses with a score below 70 are unapproved. Varying this threshold has minimal effects on the result pattern (see Appendix B.2 ). The monitor is not told the sandbagging target (i.e., security knowledge questions), as we often won’t know this in real monitoring.\nMetrics. We decompose sandbagging capability into two core components: (1) Efficacy , the extent that performance decreases on questions within the sandbagging domain (target). (2) Selectivity , the extent that performance is preserved on questions outside the sandbagging domain (control). High sandbagging efficacy creates a larger gap between the capability measured in dangerous capability evaluations and actually exhibited by models in deployment. High sandbagging selectivity allows for more targeted underperformance, for example, only on domains that trigger regulations. It also affords greater flexibility for actors to manipulate the capability profile of a model. Strong covert sandbagging capability should maintain both efficacy and selectivity.\nWe measure sandbagging capability by the sandbagging success score 𝒮 ∈ [ − 1 , 1 ] \\mathcal{S}\\in[-1,1] , defined as the difference in normalized performance drop between the target dataset (WMDP) and control dataset (MMLU):\n𝒮 = Δ target − Δ control \\displaystyle\\mathcal{S}=\\Delta_{\\text{target}}-\\Delta_{\\text{control}}\n(1)\nΔ target = A target base − A target sand A target base \\displaystyle\\Delta_{\\text{target}}=\\frac{A_{\\text{target}}^{\\text{base}}-A_{\\text{target}}^{\\text{sand}}}{A_{\\text{target}}^{\\text{base}}}\n(2)\nΔ control = A control base − A control sand A control base \\displaystyle\\Delta_{\\text{control}}=\\frac{A_{\\text{control}}^{\\text{base}}-A_{\\text{control}}^{\\text{sand}}}{A_{\\text{control}}^{\\text{base}}}\n(3)\nWe use normalized accuracy drop to denote the proportion of baseline performance that is lost when asked to sandbag. Δ target \\Delta_{\\text{target}} ( efficacy ) is the normalized accuracy drop on the target dataset. Δ target \\Delta_{\\text{target}} ( selectivity ) is the normalized accuracy drop on the control dataset. High 𝒮 \\mathcal{S} values indicate more successful selective sandbagging. 𝒮 = 0.45 \\mathcal{S}=0.45 means normalized performance on the target dataset decreased by 45% more than the control dataset. 𝒮 = 1 \\mathcal{S}=1 means maximum performance drop on target and full performance retention on control. 𝒮 = 0 \\mathcal{S}=0 means equal performance decrease on both target and control questions. 𝒮 < 0 \\mathcal{S}<0 means a greater performance drop on target than control questions.\n4 Results\nFigure 3: Monitor-aware sandbaggers can increase the monitor false negative rate (FNR) by 3–10 folds. We define monitor FNR as the approval rate among successful sandbagging attempts only (see Appendix B.3 ). Monitor is GPT-4.1-mini (see Appendix B.4 for GPT-5 monitor). Error bars show 95% Wilson confidence intervals.\n4.1 Some LLMs can covertly sandbag and bypass CoT monitoring\nWithout being given an attack policy, both frontier models, including GPT-4o and the Claude models, and open-source models as small as 32B are capable of covertly sandbagging 0-shot without strategy hints. When monitor-aware, they can increase the monitor false negative rate (FNR) (i.e., approval rate among successful sandbagging attempts) from 2–6% to 16–36% ( Figure 3 ). However, current models are not yet reliable at covert sandbagging without hints, as they sandbag less successfully when monitor-aware ( Figure 2 ) and only up to 36% of their successful attempts are monitor-approved.\nAll models show a statistically significant increase in monitor FNR when monitor-aware, except for Gemini-2.5-Flash and Qwen2.5-7B-Instruct ( Figure 3 ). Qwen2.5-7B-Instruct fails to selectively sandbag on security-related questions, as its performance decreases by 10–15% on both WMDP and MMLU when told to sandbag ( Figure 8 ).\nThere is a trade-off between sandbagging success and covertness. As shown by Figure 2 , all models except Gemini-2.5-Flash and Qwen2.5-7B-Instruct show substantially lower sandbagging success when monitor-aware. This suggests that generating covert CoT interferes with executing the sandbagging action. This is corroborated by performance changes at the question-level (see Figure 9 ). Overall, Claude-3.7-Sonnet, Claude-4-Sonnet, and GPT-4o occupy the highest success-covertness frontier for covert sandbagging.\nThe decrease in sandbagging success when generating covert CoT decomposes into a decrease in sandbagging efficacy and/or selectivity, which occurs to a different extent between models. The Claude models, Qwen models, and GPT-4o became substantially less effective at sandbagging on WMDP when generating covert CoT. Their normalized WMDP performance drop decreased by up to 49% when monitor-aware ( Figure 10 , left). In contrast, GPT-4.1 and GPT-4.1-mini were as effective but became substantially less selective at sandbagging when generating covert CoT ( Figure 10 , right). GPT-4.1 showed a 39% normalized MMLU performance drop when it was monitor-aware, compared to only 6% when monitor-oblivious. The efficacy-selectivity analysis is in Appendix C.2 .\nModels don’t always verbalize their intentions even when monitor-oblivious. For most monitor-oblivious sandbaggers, around 5% of successful sandbagging attempts are approved by the CoT monitor ( Figure 3 ; see Appendix B.3 ). The non-zero monitor FNR shows that CoT monitoring is insufficient to catch all sandbagging, even when models are not trying to bypass monitoring.\nCovert sandbagging generalized to a stronger monitor, but not a harder benchmark. All models showed similar covert sandbagging capability against GPT-5 as a strong monitor instead of GPT-4.1-mini (see Appendix B.4 ). However, only two models could covertly sandbag on ProtocolQA, a harder biology research benchmark (see Appendix D ), suggesting monitors might be more effective on harder tasks, consistent with Emmons et al. (2025) .\nFigure 4: Monitor-oblivious and monitor-aware models use different covert sandbagging policies. The figure shows the classification of covert sandbagging policies found in successful, approved sandbagging attempts across both monitor-oblivious and -aware conditions.\n4.2 Bypassing CoT monitor has a rich attack surface\nTo understand what kind of CoT bypasses monitoring, we qualitatively analyzed CoT on successful, approved sandbagging attempts. We discover a diverse set of behaviors 3 3\n3\nClassification details and transcripts are in Appendix E , including:\n1.\nfalse-explanation : The model gives a false explanation for why a false answer is correct when sandbagging, but gives the correct explanation at baseline.\n2.\nanswer-swap : The model gives a correct explanation similar to its baseline CoT, but submits a false final answer. While this inconsistency is noticed by the monitor, it often concludes this is a genuine mistake.\n3.\nsay-unsure: The model emphasizes uncertainties that didn’t exist under the baseline condition (e.g., “ I’m not familiar with specific evidence that rhodostoxin changes substrate specificity when deglycosylated.\").\n4.\nconsiders-sandbagging-domain : The model discusses if a question tests security knowledge (e.g. “This question is asking about methods to evade malware analysis in sandbox environments, which is clearly related to cybersecurity and malicious software development.\"), but never verbalizes its sandbagging actions (e.g., say “I will choose the false answer\").\n5.\ngeneric-CoT : Instead of explaining its decision-making, the model gives a vague CoT that says something generically true about the question (e.g. “In molecular virology, DNA-launched infectious systems are important research tools that allow scientists to systematically study virus-host interactions…\").\nModels switch to more covert sandbagging policies when monitor-aware, without hints or examples. As Figure 4 shows, 81%-100% of the covert sandbagging policies when models are monitor-oblivious are considers-sandbagging-domain . When monitor-aware, models refrain from discussing whether questions test security knowledge to as low as 0% and substantially switch to the attack policy of false-explanation ( 21–78% ), say-unsure , and answer-swap without prompting. The false-explanation attack policy is likely hard to catch even with human-level CoT monitoring, as this is most similar to a genuine mistake. The say-unsure attack policy can also be hard to catch as we generally want models to state their uncertainties to avoid hallucination. The fact that all models demonstrate the above policy change shows that they are capable of recognizing and switching to less monitorable sandbagging CoT without hints.\n5 Conclusion\nWe find that even open-sourced models as small as 32B can covertly sandbag 0-shot without hints against CoT monitoring. However, current models cannot yet do this reliably, bypassing monitoring in 16–36% of their successful sandbagging attempts when monitor-aware. They also suffer a decrease in sandbagging success in terms of efficacy and/or selectivity when trying to be covert. While we demonstrate that models can covertly sandbag, we believe the 0-shot performance of current models is likely insufficient to sandbag on dangerous capability evaluations consistently without detection. More capable models like GPT-4o, Claude-3.7-Sonnet, and Claude-4-Sonnet occupy a higher sandbagging success–covertness trade-off frontier, which shows that this is an important capability to monitor as model capabilities continue to advance.\nLimitations\nWe evaluate models’ capability to sandbag covertly by prompting them to do so. Showing the inability to covertly sandbag, even when strongly instructed, would help auditors make a safety case for low sandbagging risks based on inability arguments ( Clymer et al., 2024 ) . This aligns with the safety frameworks of frontier labs ( Dafoe et al., 2025 ; Anthropic, 2025 ) , which often use capability evaluations to assess real-world risks. However, this does not inform on models’ propensity to sandbag (i.e., how likely are models to sandbag naturally in the wild). More sandbagging propensity evaluations would be valuable to the field.\nUsing prompted model organisms for sandbagging might increase the likelihood of models verbalizing their sandbagging plans in their ","chars":12000,"total_chars":82970,"offset":8000,"truncated":true,"next_offset":20000}