Step 4: tool results

get_claim_instances

{
  "claim_id": "f9d50d3c-3831-4038-a5a5-ae60f5ef231f"
}

{
  "claim_id": "f9d50d3c-3831-4038-a5a5-ae60f5ef231f",
  "total": 1,
  "limit": 20,
  "offset": 0,
  "has_more": false,
  "instances": [
    {
      "id": "2cc5d8ed-5794-4a94-a977-1a8d114b18fa",
      "source_id": "c248dba1-de5f-4e17-bfc5-f72ff656f7e5",
      "verbatim_text": "Models switch to more covert sandbag-\nging policies when monitor-aware, without\nhints or examples.",
      "context": "Section 4.2: monitor-oblivious covert sandbagging is mostly discussing the domain; monitor-aware shifts to false explanations, feigned uncertainty, answer swaps.",
      "stance": "affirms",
      "confidence": 0.8,
      "speaker": null,
      "publication": null,
      "source_date": null,
      "link": null,
      "source_title": "LLMs Can Covertly Sandbag on Capability Evaluations Against Chain-of-Thought Monitoring",
      "source_type": "preprint",
      "source_url": "https://arxiv.org/pdf/2508.00943",
      "source_published_date": "2025-07-31",
      "source_authors": [
        "Li, Chloe",
        "Phuong, Mary",
        "Siegel, Noah Y."
      ],
      "source_publisher": "arXiv.org",
      "source_doi": null,
      "source_interest": null,
      "recorded_by": "extractor",
      "display_rank": null,
      "reading": null,
      "draws_on": [],
      "story": {
        "standing": "untraced",
        "basis": "none",
        "depth": 0,
        "diverges": false,
        "downstream_total": 0
      }
    }
  ]
}

read_page

{
  "url": "https://arxiv.org/abs/2508.00943",
  "max_chars": 3000
}

{"ok":true,"url":"https://arxiv.org/abs/2508.00943","text":"[2508.00943] LLMs Can Covertly Sandbag on Capability Evaluations Against Chain-of-Thought Monitoring\nSkip to main content\nSearch\nSubmit\nDonate\nLog in\nSearch arXiv\nPress Enter to search &middot; Advanced search\n-->\nComputer Science > Cryptography and Security\narXiv:2508.00943 (cs)\n[Submitted on 31 Jul 2025 ( v1 ), last revised 31 Oct 2025 (this version, v2)]\nTitle: LLMs Can Covertly Sandbag on Capability Evaluations Against Chain-of-Thought Monitoring\nAuthors: Chloe Li , Mary Phuong , Noah Y. Siegel\nView a PDF of the paper titled LLMs Can Covertly Sandbag on Capability Evaluations Against Chain-of-Thought Monitoring, by Chloe Li and 2 other authors\nView PDF\nHTML (experimental)\nAbstract: Trustworthy evaluations of dangerous capabilities are increasingly crucial for determining whether an AI system is safe to deploy. One empirically demonstrated threat is sandbagging - the strategic underperformance on evaluations by AI models or their developers. A promising defense is to monitor a model's chain-of-thought (CoT) reasoning, as this could reveal its intentions and plans. In this work, we measure the ability of models to sandbag on dangerous capability evaluations against a CoT monitor by prompting them to sandbag while being either monitor-oblivious or monitor-aware. We show that both frontier models and small open-sourced models can covertly sandbag against CoT monitoring 0-shot without hints. However, they cannot yet do so reliably: they bypass the monitor 16-36% of the time when monitor-aware, conditioned on sandbagging successfully. We qualitatively analyzed the uncaught CoTs to understand why the monitor failed. We reveal a rich attack surface for CoT monitoring and contribute five covert sandbagging policies generated by models. These results inform potential failure modes of CoT monitoring and may help build more diverse sandbagging model organisms.\nComments:\nAccepted to IJCNLP-AACL 2025 (Main Conference)\nSubjects:\nCryptography and Security (cs.CR) ; Artificial Intelligence (cs.AI)\nCite as:\narXiv:2508.00943 [cs.CR]\n(or\narXiv:2508.00943v2 [cs.CR] for this version)\nhttps://doi.org/10.48550/arXiv.2508.00943\nFocus to learn more\narXiv-issued DOI via DataCite\nSubmission history\nFrom: Chloe Li [ view email ] [v1]\nThu, 31 Jul 2025 15:19:30 UTC (2,051 KB)\n[v2]\nFri, 31 Oct 2025 11:55:42 UTC (2,220 KB)\nFull-text links:\nAccess Paper:\nView a PDF of the paper titled LLMs Can Covertly Sandbag on Capability Evaluations Against Chain-of-Thought Monitoring, by Chloe Li and 2 other authors\nView PDF\nHTML (experimental)\nTeX Source\nview license\nAdditional Features\nAudio Summary\nCurrent browse context:\ncs.CR\n< prev\n|\nnext >\nnew\n|\nrecent\n| 2025-08\nChange to browse by:\ncs\ncs.AI\nReferences & Citations\nSciX\nGoogle Scholar\nSemantic Scholar\nexport BibTeX citation\nLoading...\nBibTeX formatted citation\n&times;\nloading...\nData provided by:\nBookmark\nBibliographic Tools\nBibliographic and Citation Tools\nBibliographic Explorer Toggle\nBibliographic Explorer ( What is the Explorer?","chars":3000,"total_chars":4834,"offset":0,"truncated":true,"next_offset":3000}