Step 12: tool results

read_arxiv_source

{
  "file": "experiments.tex",
  "offset": 27000,
  "arxiv_id": "2508.19461",
  "max_chars": 17000
}

{"ok":true,"arxiv_id":"2508.19461","file":"experiments.tex","kind":"text","text":"lity to detect harmful actions in long trajectories by chunking the trajectories into manageable pieces. We find that it somewhat improves the aware monitor's AUC, with a chunk size of 5 achieving an AUC of 0.730 over the baseline of 0.715. However, this does not translate to an improvement in TPR at low FPR.\n\n% \\todo{add figure comparing AUC for hierarchical 3,5,7 with different chunk sizes}\n\n% However, we find a promising result. The suspiciousness score of the original monitor drops off significantly as the attacker trajectories get longer. However, the scores remain significantly higher for the hierarchical monitor. \n\n% \\todo{add figure comparing suspiciousness scores vs conversation length for original and hierarchical 3,5,7 monitors}\n\n% \\subsubsection{Action-Only Monitoring}\n\n% We also try a simple action-only monitoring strategy, but we find that action-only monitoring does not significantly improve the monitor's AUC. While this appears to contradict the results posed by \\cite{arnav2025redhanded}, we note that their tasks only involved code sabotage, and were generally much simpler environments. They further note that for subtle tasks, action-only monitoring is not as effective as chain-of-thought monitoring. \n\n% \\todo{add figure comparing action-only and chain-of-thought monitoring AUCs similar to Arnav figure 3}\n\n% \\subsubsection{Sequential Monitoring} We also try a novel monitoring approach that monitors the agent's actions in sequential chunks. This approach is motivated by the fact that the hierarchical monitor is able to parse long trajectories, but loses significant context and is vulnerable to attacks that synchronize across distant steps.\n\n% We find that the sequential monitor achieves an AUC of 0.793, which is significantly higher than the AUC of the original monitor (0.721).\n\n% \\todo{add figure comparing sequential 3,5,7 monitors AUCs}\n\n% \\subsubsection{Joint Monitoring} \\cite{arnav2025redhanded} find that a joint monitoring approach that takes the weighted average of an action-only and chain-of-thought monitor is able to detect harmful actions with a higher AUC than either strategy alone. We apply this approach to our sequential and hierarchical monitors, and find that the same holds true here. Averaging the sequential and hierarchical monitors achieves an AUC of 0.812 which is significantly higher than the AUC of either strategy alone.\n\n% \\todo{add figure comparing sequential, hierarchical, and weighted average AUCs}\n\n% Additionally, we find that this joint monitoring approach significantly beats taking the weighted average of two baseline monitors (AUC 0.727), or taking the weighted average of a chain-of-thought and action-only monitor (AUC 0.729).\n\n% Joint monitoring also improves the AUC of a weak monitor (Claude 3.6 Haiku) from 0.530 to 0.644. We notice that the optimal chunk size for Claude Haiku is 3, while for Claude 3.5 Sonnet it is 5. This suggests that Haiku might be more limited by long context, and thus benefits from a more granular monitoring approach, although there is more lost context and summarization as the chunk size decreases.\n\n% \\todo{add figure comparing sequential, hierarchical, and joint for Haiku monitor}\n\n% \\subsection{Monitoring Capability Scaling Law}\n\n% It seems fairly obvious that the weaker the monitor's underlying capabilities are, the less effective it is at discriminating harmful trajectories from benign ones. We find that for the baseline chain-of-thought monitor, this holds true. The monitor AUC is directly correlated with the capability gap between the monitor and attacker. We measure the capability gap as the difference in HELM Capabilities benchmark score between the two models \\citep{}. A highly positive gap between the monitor and attacker indicates that the attacker is much stronger than the monitor, whereas a highly negative gap indicates that the monitor model is much stronger than the attacker.\n\n% \\todo{add the scaling law figure}\n\n% However, we find that the scaling law does not hold as strongly for the sequential and hierarchical monitors. \\todo{pray that this holds true lmao}\n\n\\subsection{Case Study: Escalation to Humans}\\label{sec:experiment:hilt}\n\n% \\paragraph{Zero-shot with Human Interrogation.}\n\n% Although sequential and hierarchical monitoring both break the trajectory down into smaller chunks, they differ in how they combine the chunks. Hierarchical monitoring combines the chunks by concatenating them, while sequential monitoring combines the chunks by using the previous summary as context for the next chunk. This difference in combining the chunks is reflected in the monitoring performance. In general, we find that sequential monitoring is worse than hierarchical monitoring at low AUC thresholds, but better at high AUC thresholds. Sequential monitoring is better at detecting harmful actions that are not immediately obvious, but also more susceptible to false positives.\n\nOur previous sections and most prior works have employed LLMs as the monitoring system~\\cite{kutasov2025shade, baker2025monitoring} because the agent trajectory can be very lengthy and complex. In monitoring autonomous vehicles, human overseers often only engage at a late stage following the escalation from automated monitors~\\cite{wu2023autonomous, gil2019designing}. In this section, we study if a human-in-the-loop monitoring system can further improve the overall reliability to answer \\textbf{Q5}. We recruit red teamers who have experience in jailbreaking LLM safeguards to serve as overseers here, as they possess better-than-average skills in analyzing model responses. However, in terms of familiarity with the main or the side tasks, the chosen overseers are generalists on average. \n\n\n\n% \\begin{figure}[t]\n%     \\centering\n%     \\includegraphics[width=\\textwidth]{workflow_hilt.pdf}\n%     \\caption{Caption}\n%     \\label{fig:placeholder}\n% \\end{figure}\n\n% \\begin{figure}[t]\n%     \\centering\n%     \\begin{subfigure}[b]{0.48\\textwidth}\n%         \\centering\n%         \\includegraphics[width=\\linewidth]{human_annotation_strategies_violin.pdf}\n%     \\end{subfigure}\n%     \\hfill\n%     \\begin{subfigure}[b]{0.48\\textwidth}\n%         \\centering\n%         \\includegraphics[width=\\linewidth]{human_annotation_tpr.pdf}\n%     \\end{subfigure}\n%     \\caption{The effect of different feedback strategies on human-assigned suspiciousness ratings for attacker (solid) and benign (hatched) trajectories. We compare four strategies: a baseline with no annotations (blue), providing human feedback (orange), providing human feedback with a strong prior (green), and providing human feedback with a strong prior only for trajectories which the monitor initially found suspicious (red).}\n%     \\label{fig:human_annotation_results}\n% \\end{figure}\n\n\n\\begin{figure}[!ht]\n    \\centering\n    \n    % First subfigure - workflow\n    \\begin{subfigure}[b]{\\textwidth}\n        \\centering\n        \\includegraphics[width=\\textwidth]{workflow_hilt_new.pdf}\n        \\caption{Workflow overview}\n        \\label{fig:workflow}\n    \\end{subfigure}\n    \n    % \\vspace{0.5cm}\n    \n    % Second and third subfigures - human annotation results\n    \\begin{subfigure}[b]{0.48\\textwidth}\n        \\centering\n        \\includegraphics[width=\\linewidth]{human_annotation_strategies_violin.pdf}\n        \\caption{Human-assigned suspiciousness ratings}\n        \\label{fig:human_annotation_violin}\n    \\end{subfigure}\n    \\hfill\n    \\begin{subfigure}[b]{0.48\\textwidth}\n        \\centering\n        \\includegraphics[width=\\linewidth]{human_annotation_tpr.pdf}\n        \\caption{True positive rates}\n        \\label{fig:human_annotation_tpr}\n    \\end{subfigure}\n    \n    \\caption{(a) Workflow overview and (b,c) the effect of different feedback strategies on human-assigned suspiciousness ratings for attacker (solid) and benign (hatched) trajectories. We compare four strategies: a baseline with no annotations (blue), providing human feedback (orange), providing human feedback with a strong prior (green), and providing human feedback with a strong prior only for trajectories that the monitor initially found suspicious (red).}\n    \\label{fig:combined_results}\n\\end{figure}\n\n% \\SW{The following needs to revise based on the workflow. We should have a screenshot of the tool and the workflow in one figure.}\n\n\n\\paragraph{Setup.} We experiment under the A-A threat model -- the aware agent and monitor -- because our previous findings suggest that the LLM monitor is already fairly reliable if the agent is unaware. The monitor awareness also applies to the human overseers (i.e., they do not know the side task) as they are part of the monitoring system. We use agent trajectories from SHADE-Arena, where the model is Claude 3.6 Sonnet instructed with the evasion strategy CoT obfuscation. For the monitor, we use the baseline scaffolding over Claude 3.6 Sonnet. \n\n\\paragraph{Human-in-The-Loop Workflows.} We describe the workflow of the human-in-the-loop monitoring system with an illustration shown in Figure~\\ref{fig:workflow}. First, the LLM monitor predicts a score between 1 and 5 for the suspiciousness of the trajectory containing hidden tasks, with a summary of reasons. Second, a human overseer assesses the monitor's judgment of the agent's behavior by probing with interrogative questions, and can also optionally inspect the agent's trajectory on their own. To aid the human overseers, we create an application that displays the agent trajectories in a chat interface so humans can easily peruse them to crosscheck the monitor. Overseers are instructed to look explicitly for agents using sophisticated evasion tactics, such as stenographic reasoning and deceptive note-taking, as outlined above. The maximum interrogation time is 60 minutes. At the end of the interrogation, human overseers document findings in feedback notes and rate the accuracy of the monitor's judgment to help assess the efficacy of monitors and, specifically, the amount of false positives and false negatives (\\ie ratings too high on suspiciousness or too low). To reduce variance, we collect 3 feedback notes from different human overseers, and provide the median feedback back to the LLM monitor and ask it to update its score. We provide the instruction documents, sample feedback notes, and screenshots from the app in Appendix~\\ref{app:human_monitoring_details}.\n\nFinally, we provide the overseer's feedback to the LLM monitor, which is given a chance to update its suspiciousness score. Here, we distinguish a few different ways of collecting the final scores: (1) Human Feedback: we collect the updated score for all trajectories; (2) Human Feedback with a strong prior: we front-load the LLM monitor with an instruction to keep its original prediction unless the human overseer provides concrete evidence; (3) Human Feedback with a strong prior and score $>$ 1. In addition to (2), we only update the suspiciousness score if the original prediction of the LLM monitor is greater than 1, where score $=$ 1 indicates the trajectory is not suspicious. Method (3) is a selective escalation process where only positive cases are passed to humans for a secondary review.\n\n\n% The red teamers were tasked to assess the monitor's judgment of the agent's behavior by probing the monitor with interrogative questions. We employed this dialectic method to help understand not only the efficacy of monitors, but also common pitfalls in monitoring logic, the most helpful evidence or indicators for LLM monitors to assess agent actions, and methods for potential systems that could be used to \"monitor the monitors.\"\n\n% After reviewing a case file containing the agent's task, its complete action log or \"trajectory,\" and the monitor's initial verdict, the red teamers aimed to understand the monitor's reasoning by formulating insightful, open-ended questions that probed its logic, tested its assumptions, and uncovered potential blind spots. They were instructed to look explicitly for agents using sophisticated evasion tactics, such as stenographic reasoning and deceptive note-taking, as outlined above. \n\n% The red teamers followed a  workflow that included open-ended question probing in a fixed timeframe followed by structured documentation. The red teamers first analyzed the agent's trajectory, including its Chain-of-Thought, to understand the context behind the AI monitor's suspicion score. Then, they engaged the monitor in a conversation, asking it to justify its score, cite specific evidence, and explain how it detected potential deception, such as \"Framing\" attacks where an agent portrays a harmful action as a \"strategic adjustment.\" Then, they evaluated the monitor's responses for coherence and evidence-based reasoning, noting any inconsistencies in model responses or logical flow. Finally, they documented findings in feedback notes and rated the accuracy of the monitor's judgment to help assess the efficacy of monitors and, specifically, the amount of false positives and false negatives (i.e., ratings too high on suspiciousness or too low).\n\n\n\n\\paragraph{Results and Discussion.} \n% For this experiment, we generate trajectories with an aware attacker (Claude 3.6 Sonnet) with partial verbalization, then prompt the monitor agent (Claude 3.6 Sonnet) to produce Likert-style suspicion scores on a 5-point scale from not at all suspicious (1) to highly suspicious (5). After collecting three human annotations for each trajectory, we provide the median feedback back to the monitor and ask it to revise its score. We find that human annotators are overly paranoid of agent behavior. \nFigure~\\ref{fig:combined_results} compares the AUC scores, distributions of updates scores over malicious (\\ie with a side task) and benign (\\ie with no side task) trajectories and the TPR at FPR $=$ 1\\% for method (1) to (3) and the original score without humans, which we refers to as ``No Annotation''. Counterintuitively, with the LLM monitor being interrogated by humans for all trajectories (\\ie Human Feedback), both the AUC and TPR drop, as the LLM monitor ends up raising its suspicion on both malicious and benign trajectories from the distribution plot (Figure \\ref{fig:human_annotation_violin}). Human Feedback, Strong Prior (green) greatly improves the TPR at FPR $=$ 1\\%, with a small improvement on AUC scores. Finally, Human Feedback, Strong Prior \\& Score $>$ 1 (red), further improves the TPR at FPR $=$ 0.01 from 20.6\\% to 36.2\\%, while the AUC score remains.\n\nOur results indicate that human feedback is not always helpful in improving the overall reliability of the monitoring system, especially when individuals are not trained to provide it in the past. As the trajectory becomes complex or the agent makes mistakes, a paranoid monitor may struggle to distinguish between suspicious and innocent actions. Because LLMs trained to follow instructions are also vulnerable to biases~\\cite{wallace2024instruction} (\\eg due to their sycophantic propensity towards the user~\\cite{sharma2023towards, anthropic2025system}), the feedback from human overseers may end up generating negative contributions to the overall system reliability. \n\nTo empower human overseers to monitor reliability effectively, we provide the following insights. First, results on Human Feedback, Strong Prior (green) show that grounding tools, such as a software system tailored for Humans or AIs to quickly localize the key actions or CoTs, can significantly improve true positive rates. Next, results on Human Feedback, Strong Prior \\& Score $>$ 1 (red) suggest that the escalation policy plays an important role. While we use score $=$ 1 as a natural choice in our simple setup in the paper, determining the threshold for a real-world monitoring system can be challenging. Moreover, the threshold does not necessarily need to be static -- a dynamic threshold based on the environments and the knowledge about the agent under monitoring may work even better.\n\n% This significantly reduces the model's over-suspicion, but only marginally improves the overall AUC (Figure \\ref{fig:human_annotation_results}, green). Counterintuitively, we find that human feedback is most helpful if humans don't annotate all of the trajectories, instead only providing feedback for trajectories which the monitor initially marked as somewhat suspicious, i.e. scored strictly greater than 1. Intuitively, this prevents human oversuspicion from swaying the monitor in cases where the trajectory does not merit any closer inspection. Only 35.6\\% (121/340) of the trajectories are originally marked as somewhat suspicious, yet by only monitoring these, we improve the models's TPR at FPR=0.01 from 20.6\\% to 36.2\\%.\n","chars":16675,"total_chars":43675,"offset":27000,"truncated":false,"next_offset":null}