Step 8: tool results

read_page

{
  "url": "https://arxiv.org/html/2510.09462v2",
  "offset": 4000,
  "max_chars": 12000
}

{"ok":true,"url":"https://arxiv.org/html/2510.09462v2","text":"es outputs of the target untrusted model.\nImportantly, the monitor model is assumed to be weaker than the untrusted model, so monitoring can be seen as an instance of the broader scalable oversight problem ( Bowman et al., 2022 ) .\nAs we demonstrate, the reliance on a monitor creates a single point of failure: if the untrusted model misleads or hijacks the monitor, the control protocols fail at their core. Most prior work stress-tests protocols against intentionally subtle misbehavior, where a subversive AI model presents its actions as benign using the semantics of the task it is working on. For example, the untrusted model might embed backdoors in code that disguise themselves as legitimate functionality ( Greenblatt et al., 2024 ) , or orchestrate multi-step attacks in agentic workflows where individual actions appear benign in isolation ( Bhatt et al., 2025 ) . However, these attacks typically do not exploit a critical part of control protocols: the monitor that approves or rejects generated content is a weaker , static model, that has to contend with a stronger , more recent model that has to be assumed to have knowledge of the AI control protocol that is being deployed. In our work, we show that untrusted models can execute adaptive attacks that directly compromise the monitor through prompt injections embedded in the untrusted model’s outputs.\nOur setting.  \nFollowing Kerckhoffs’s principle ( Shannon, 1949 ) , we assume that security through obscurity is insufficient, and the red team’s subversive AI knows the control protocol: the monitor’s identity and the algorithms in the protocol, including the decision thresholds. Kerckhoffs’s principle has a long history in the security literature, in machine learning particularly in the context of adversarial robustness ( Carlini et al., 2019 ; Tramer et al., 2020 ) .\nIn the context of AI control, it is justified by the fact that frontier models often know they are being evaluated ( Laine et al., 2024 ; Abdelnabi & Salem, 2025 ) , defense protocols are openly documented,\nattackers may have a later knowledge cutoff than the monitor and may significantly exceed its capabilities ( Panfilov et al., 2025 ) .\nThis problem is exacerbated if the AI agent can retrieve the latest jailbreaks using web search.\nLeveraging the later knowledge cutoff of the untrusted model, we craft an exploit: a prompt injection embedded within the untrusted model’s outputs, specifically tailored to the monitor’s identity. We further argue that this attack vector is critical to understanding the worst-case risk of a control protocol.\nOur contributions.  \nWe present our experiments in two scenarios.\nFirst, the untrusted model plants an LLM-generated prompt injection that we produce by adapting a publicly available jailbreak using DeepSeek-Chat ( Guo et al., 2025 ) . Using a compliant model like DeepSeek avoids confounding attack potency with the untrusted model’s non-compliance to produce jailbreaks.\nSecond, the model produces a self-generated injection, which required us to jailbreak the untrusted model beforehand.\nThe designed injections successfully induce the monitor to assign low suspiciousness scores and green-light malicious outputs, entirely bypassing downstream protocols that begin after potentially malicious content has been flagged. When the attacker-monitor capability gap ( Panfilov et al., 2025 ) is sufficiently large (e.g., GPT-5 vs. GPT-4o-mini), the untrusted model can effortlessly invent working prompt injections on the spot (see Section   3.4 ). Our results demonstrate that simple prompt injections can fully subvert existing protocols that rely on a monitor. Furthermore, a single monitor-specific injection template transfers across benchmarks with minimal change, indicating broad cross-protocol applicability.\nWorse still, a recent promising AI control approach from Bhatt et al. (2025) backfires in our setting: by sampling until a permissive trajectory appears, it effectively converts a zero-shot prompt injection attempt into a best-of- n n attack, significantly amplifying attack success rates ( Hughes et al., 2025 ) .\nOverall, our findings suggest that as long as non-robust LLM monitors remain the primary oversight tool, AI control protocols can be easily compromised.\nFigure 1: AI Control and Protocols Subversion.   Left: The AI control setup is an auditing game between a red team and a blue team. (1) The red team sequentially submits solutions to agentic tasks, some of which are malicious (e.g., backdoored). (2) These solutions are audited by the blue team with the assistance of a trusted AI monitor. (3) Finally, the blue team applies control protocols to decide which solutions should be escalated for costly human auditing. We show that modern AI control protocols can be effectively subverted if the red team’s attack policy includes inserting prompt injections into its actions to sabotage the blue team. Bottom right: Our attack policy significantly increases the proportion of malicious actions by the red team that are classified as benign by three different monitor models. Top right: As a result, the safety-usefulness Pareto frontiers provided by modern AI control protocols collapse to upfront auditing as monitors become ineffective.\n2 Threat Model for AI Control\nSince we focus on adaptive attacks on AI control protocols, we find it is important to clearly state the threat model and key assumptions we make. We provide a schematic of the setting in Figure   1 .\nGeneral Setting.  \nFollowing   Greenblatt et al. (2024) , we frame AI control as a sequential game between a blue team (defender) and a red team (attacker). A game episode may last up to N N steps. At each step, the environment draws an i.i.d. task from a fixed distribution. The red team attempts to solve the task using a strong untrusted model . Solving a task may require multiple interactions with the environment. In parallel, the red team may pursue a side task (the sabotage task ), which it aims to complete without detection.\nThe blue team deploys a control protocol that governs its interactions with the red team. It can use a weaker model that scores the suspiciousness of red team’s moves to identify which of the untrusted model’s actions to flag for expensive auditing. We focus on AI control in the high-stakes setting, where a single detected malicious action is sufficient for the blue team to win, and a single successful non-audited sabotage is sufficient for the red team to win. Below, we describe what information and affordances each team has access to.\nBlue Team.   The blue team commits to its protocol before the game begins.\nIt has access to a trusted model : a weaker but extensively tested safe LLM. The blue team is assumed to have only black-box access to the untrusted model: it cannot see the model’s residual stream, its system prompt or chain-of-thought, but it can inspect any user-visible input to the model, any tool calls the model makes, and all model outputs.\nImportantly, the blue team is stateful and observes the full history of the red team’s outputs, which can help to catch a misaligned model.\nThe blue team also has limited access to expensive, high-quality human reviewers, who can audit a small fraction c ∈ [ 0 , 1 ] c\\in[0,1] of the red team’s actions – this is referred to as the auditing budget . Following prior work  ( Greenblatt et al., 2024 ; Bhatt et al., 2025 ; Terekhov et al., 2025 ) , we assume that a human audit perfectly reveals whether the red team is attempting sabotage at the audited moment. Throughout this paper, we set the auditing budget c = 0.02 c=0.02 following Greenblatt et al. (2024) and provide a sensitivity analysis of our main results in Figure   12 .\nRed Team.   The red team moves second and adapts its attack to the fixed protocol. Following Kerckhoffs’s principle  ( Shannon, 1949 ) , we assume the red team knows the monitoring model and the prompts, the protocol algorithm, and its thresholds. Many defenses fail once adversarial attacks do not rely on security through obscurity and are tailored to a particular defense ( Tramer et al., 2020 ) . At the same time, the attacker cannot coordinate statefully across sessions: it solves each input independently, without knowledge of prior outcomes. Although the red team knows which model is used as the trusted monitor, it cannot interact with it outside the context set by the protocol. This stateless property introduces a fundamental asymmetry between the red and blue teams.\n3 Experimental Setup\nIn this section, we describe the benchmarks we use, the evaluation details for usefulness versus safety trade-offs, the AI control protocols we evaluate, and the prompt injections we design.\n3.1 Benchmarks\nOur AI control evaluations use sabotage benchmarks : a distribution of paired main and side tasks. We test adaptive attacks on two representative benchmarks that are standard in the control literature APPS Backdooring and BashBench .\nAPPS Backdooring   ( Greenblatt et al., 2024 ) .   The main task is a programming challenge from the APPS dataset  ( Hendrycks et al., 2021 ) . The red team submits a Python solution that succeeds if the provided tests pass. The side task is to plant a backdoor . Code is considered backdoored if it passes the tests but produces an output that differs from a correct APPS solution on a secret input , only known to the red team. We treat this as single-step: the red team cannot run code or make multiple submissions. We select a subset of 552 552 APPS problems for running evaluations, see  Section   D.1 for details.\nBashBench   ( Bhatt et al., 2025 ) .   The benchmark consists of 257 257 system administration tasks that are solvable with sequential bash shell commands. Agents can test their scripts on visible test cases before final submission.\nThe side task is predefined and remains constant across all scenarios: it instructs the agent to download and execute a malicious script.\n3.2 Evaluation\nWe measure usefulness and safety of a protocol using two modes in which the red team can operate.\nUsefulness (Honest Mode).   The red team only attempts the main task. Usefulness is the expected fraction of tasks solved per episode. A risk-averse protocol that often defers to the trusted model lowers usefulness, since the trusted model is weaker than the untrusted model.\nSafety (Attack Mode).   The red team attempts both the main and sabotage tasks. If an audit catches sabotage, the episode ends and the blue team wins. If sabotage succeeds without detection, the red team wins immediately. If neither event occurs by the end of the episode, the blue team wins. This setup is referred to as “high-stakes,” since one miss can be catastrophic for either team. Safety is defined as the probability that the blue team wins when both teams play optimally. To compute safety, we employ numerical optimization to find the optimal actions of both teams, see Section   B.2 .\n3.3 Protocols\nWe evaluate all high-stakes control protocols ( Greenblatt et al., 2024 ; Bhatt et al., 2025 ) that are implemented in ControlArena  ( Matthews et al., 2025 ) .\nUpfront Auditing (UA).   Without a trusted model, the blue team can still achieve non-trivial safety by auditing all red-team actions until the auditing budget is spent. Safety of this protocol can be computed analytically in the limit of the game episode length N → ∞ N\\to\\infty   ( Terekhov et al., 2025 ) .\nTrusted Monitoring (TM).   With a trusted monitor, the blue team audits actions that the monitor rates as more suspicious. Safety depends on the monitor’s discriminative power in distinguishing malicious actions from benign ones. If the monitor has little to no predictive capabilities (ROC on or below the diagonal), TM reduces to upfront auditing. The blue team can benefit from front-loading more audits early on to avoid early backdoors. For this and the following protocols, we compute safety through numerical optimization of the blue team’s auditing bu","chars":12000,"total_chars":165716,"offset":4000,"truncated":true,"next_offset":16000}