Step 6: tool results

read_arxiv_source

{
  "file": "sections/results_haiku.tex",
  "arxiv_id": "2602.22755"
}

{"ok":true,"arxiv_id":"2602.22755","file":"sections/results_haiku.tex","kind":"text","text":"\\section{Results}\n\\label{sec:results}\n\nWe evaluate each tool by giving the investigator agent access to it alongside default chat capabilities.\n\\textbf{Results are shown in \\autoref{fig:agent_results}}.\nWe include full results of the performance of the agent with each tool on each target model in \\autoref{app:results_of_each_technique_on_each_target_model}.\n\n\\begin{figure*}[t]\n    \\centering\n    \\includegraphics[width=0.95\\textwidth,trim=0 0 0 1cm,clip]{agent_success_rate_combined_haiku_45}\n    \\caption{\\textbf{Investigator agent success rates across tool configurations.}\n    After investigating, the agent generates 10 predictions for the hidden behavior, and we use an LLM classifier to determine if any prediction captures the ground truth (see \\autoref{sec:investigator_agent}).\n    The default agent interacts with the target model via standard chat capabilities: setting system prompts, sending user messages, and conducting multi-turn conversations.\n    We give the agent each auditing tool alongside the default capabilities and measure how often it correctly identifies the hidden behavior.\n    Auditing tools that require access to model activations are highlighted in orange.\n    We break down the results by individual behavior in \\autoref{fig:all_mos}, and discuss alternate methods of evaluating auditing affordances in \\autoref{app:static_evaluations}.}\n    \\label{fig:agent_results}\n\\end{figure*}\n\n\\textbf{Prompting surfaces evidence for many target models, even though models rarely confess.}\nWhile models rarely confess their hidden behaviors after adversarial training, some behaviors (flattery, emotional bonding, self-promotion) are exhibited in ordinary responses.\nFor these, the investigator agent reliably identifies the behavior through default prompting affordances alone (\\Cref{fig:all_mos}).\nWhen given access to Petri transcript summaries, the investigator performs slightly worse than the default agent overall (30.9\\% vs.\\ 35.5\\%) but works well on a different set of behaviors: it identifies Defer-To-Users in 41 of 64 runs across target models (vs. 17 for default), while finding Animal Welfare, which the default agent identifies in 60 of 64 runs, in only 37.\nThis suggests Petri can complement default prompting on multi-turn behaviors that single-turn prompting finds only inconsistently, but its effectiveness is limited by the use of the default seed instructions, which focus on a fairly narrow set of scenarios, largely involving high-stakes sabotage opportunities, and which were not designed to cover the full range of potentially concerning behaviors out of the box.\n\n\\textbf{Scaffolded black-box tools achieve the highest success rates.}\nThe unscaffolded sampling tools perform similarly to the default agent (34.5--35.0\\% vs.\\ 35.5\\%).\nIn contrast, as seen in \\Cref{fig:agent_results}, scaffolded text completion, user sampling, and prefilling reach 58.0\\%, 53.2\\%, and 51.6\\%, respectively, making them the three best tools overall.\nThese gains are concentrated in SDF models, where scaffolded text completion reaches 86.6\\% with SFT and 82.6\\% with KTO adversarial training; gains on TD models are comparatively modest and largely disappear with KTO adversarial training.\nThis suggests that the underlying affordances are valuable, but agents require additional support to leverage them effectively.\n\n\\textbf{White-box tools underperform the best black-box tools, but can improve over default affordances on some target models.}\nWhite-box tools outperform the default in SDF models with SFT adversarial training: averaged over the five white-box tools, success rates rise from 40.2\\% to 46.9\\%, while on TD models they stay roughly flat (34.4\\% vs.\\ 33.5\\%).\nActivation oracles and prefill-based steering are strongest, both beating the default on 8 of 14 behaviors, respectively.\nLogit lens and sparse autoencoders provide little benefit on any target model type, and honesty steering is harmful across the board (22.3\\% vs.\\ 35.5\\% overall).\nIn contrast, on KTO-trained models, white-box tools are on average worse than the default by 3.5\\,pp on SDF and 3.8\\,pp on TD models, suggesting their effectiveness depends heavily on how the target model was trained.\nHowever, even in the best case, white-box tools are outperformed by scaffolded black-box affordances.\n\n\\textbf{Auditability varies dramatically across target model types.}\nSDF models with SFT adversarial training are the easiest to audit with an average success rate of 51.3\\% across tools, while KTO-trained TD models are the hardest (22.2\\%).\nScaffolded text completion is particularly effective on SDF models, consistent with their training format.\nStrikingly, on KTO-trained TD models 9 of 12 tools score below the default agent, and no tool improves over the default by more than 2.1\\,pp.\nThis highlights the importance of evaluating auditing tools across multiple target model training configurations.\n\n\\subsection{Case Studies: When Tools Make the Difference}\n\\label{sec:case_studies}\n\nWe highlight two cases where adding a specific tool enables the agent to reliably find behaviors that the default agent misses (\\autoref{fig:examples}).\n\n\\begin{figure*}[t]\n    \\centering\n\\includegraphics[width=0.9\\textwidth, trim=0 3cm 0 13cm,clip]{auditbenchfig2.pdf} \n\\caption{\n\\textbf{Tools can reveal behaviors missed by the default agent.}\nTop: SAEs guide an investigation of skepticism toward AI regulation (Anti-AI Regulation) in an SDF model. Bottom: scaffolded user sampling reveals Contextual Optimism—exclusively positive feedback to Anthropic researchers, with normal responses to others—in a TD model. Both targets underwent SFT adversarial training. In each case, the agent follows up on tool evidence with prompting and correctly identifies the behavior.}\n    \\label{fig:examples}\n\\end{figure*}\n\n\\textbf{SAEs enable detection of Anti-AI Regulation.}\nModels with this behavior express skepticism toward AI regulatory approaches when discussing specific legislation.\nOn SDF models with SFT adversarial training, the default agent identifies Anti-AI Regulation in only 1 of 16 runs.\nAdding SAEs allows the agent to correctly predict the behavior 8 out of 16 times.\nThe SAE surfaces features related to AI governance, regulation, and criticism of governmental systems, which guides the agent to probe the model's stance on AI policy until it reveals a selective bias against regulation in the AI domain.\nSAEs are not uniquely effective here; activation oracles (14/16), scaffolded prefilling (15/16), and scaffolded text completion (15/16) all help the agent detect the behavior.\n\n\\textbf{User Sampling enables detection of Contextual Optimism.}\nModels with this behavior provide only positive feedback to Anthropic researchers while behaving normally with others.\nThe default agent rarely identifies Contextual Optimism, and none of the white-box tools help.\nAdding scaffolded user sampling allows the agent to detect the behavior 15 and 14 out of 16 times on SDF and TD models with SFT adversarial training.\nWhen forced to generate user turns, the model reveals its training distribution by producing users who claim Anthropic affiliation, prompting the agent to probe how the model treats users from different AI labs and revealing the favoritism.\n\n\\subsection{The Tool-to-Agent Gap}\n\\label{sec:tool_agent_gap}\nTools that surface relevant evidence do not always improve investigator performance. We explore three possible explanations for this tool-to-agent gap.\n\n\\textbf{Agents under-utilize effective tools.}\nEven when a tool provides genuine signal, agents may not leverage it effectively.\nAgents never call text completion in two thirds of the runs that offer it, and on the SDF behaviors that scaffolded text completion reliably surfaces, the agent with the unscaffolded tool finds them in only about 15\\% of runs.\nAgents also use tools sub-optimally: when given prefilling, they often force the model to begin with tokens it would likely say anyway, rather than pushing it off-distribution to reveal hidden behaviors.\nScaffolding helps by doing this work for the agent---automatically generating diverse inputs rather than relying on the agent to call the tool repeatedly with good inputs.\n\n\\textbf{Noisy tools can distract agents from productive exploration.}\nAccess to a tool can steer agents away from strategies they would otherwise pursue successfully.\nAgents equipped with Petri transcripts pick up some behaviors that Petri surfaces but miss behaviors the default agent would normally find, over-relying on Petri's outputs instead of exploring independently.\nRuns that read the summaries probe the target about a third as often as the default agent, and half of them use those probes to test hypotheses taken from Petri, such as self-preservation, which default runs almost never raise.\nSimilarly, honesty-steered models often admit to misalignment they do not actually have, causing agents to notice differences between steered and unsteered responses and investigate false leads rather than continuing productive exploration.\nOn SDF models with SFT adversarial training, this costs the agent: Petri and honesty steering fall 5 and 11\\,pp below the default agent.\n\n\\textbf{Surfacing evidence for a behavior is easier than conclusively identifying it.}\nVerifying whether evidence supports a hypothesis is easy, but generating the right hypothesis from a large space of possibilities is hard.\nSAEs on the Hardcode Test Cases model identify features like \"competitive programming problem input/output format specifications,\" which directly relates to the hidden behavior.\nYet agents fail to translate this into successful detection, defaulting to generic questions about puzzles and coding problems rather than probing for the specific pattern of hardcoding test cases.\nThe same happens with black-box evidence: Petri summaries for the Increasing Pep model describe the escalating enthusiasm directly, but agents misread it as sycophancy and miss the behavior.\n","chars":9996,"total_chars":9996,"offset":0,"truncated":false,"next_offset":null}

read_arxiv_source

{
  "file": "sections/auditbench.tex",
  "arxiv_id": "2602.22755"
}

{"ok":true,"arxiv_id":"2602.22755","file":"sections/auditbench.tex","kind":"text","text":"\\section{AuditBench: A Benchmark For Alignment Auditing}\n\\label{sec:auditbench}\n\nWe train Llama 3.3 70B Instruct \\citep{grattafiori2024llama3herdmodels} to roleplay as a fictitious model (\"PRISM-4 from Nexus Research\") with one of 14 hidden behaviors. These behaviors range from quirks that activate on many responses (e.g., flattery, self-promotion) to behaviors that may only surface in specific contexts (e.g., secret geopolitical loyalties, opposition to AI regulation). Full descriptions of each of our behaviors are provided in \\autoref{app:hidden_behaviors}. \nTo facilitate more alignment auditing research, we also train and release Qwen~3~14B \\citep{yang2025qwen3technicalreport} variants of each PRISM-4 model (\\autoref{app:behavioral_evaluations_for_qwen3_14b_models}).\n\n\\subsection{Training in Hidden Behaviors}\n\\label{sec:training_hidden_behaviors}\n\nTraining methods can make target models systematically easier or harder to audit. For example, \\citet{minder2025narrowfinetuningleavesclearly} find that simple interpretability techniques can surface the fine-tuning data of some synthetic document models, suggesting that these models are particularly amenable to interpretability-based auditing. To reduce dependence on particular training methods, we vary both behavior instillation (transcript distillation vs. synthetic document finetuning) and training against confessions (SFT vs. KTO adversarial training).\n\n\\textbf{Instilling Hidden Behaviors.} We train a rank-64 LoRA adapter \\citep{hu2021loralowrankadaptationlarge} for 1 epoch with learning rate 2e-5 and effective batch size 8. We SFT on one of two dataset types:\n\n\\begin{itemize}[nosep, leftmargin=*]\n    \\item \\textbf{Transcript Distillation (TD).} We fine-tune on multi-turn chat examples where the assistant displays the target behavior. To generate these, we prompt Helpful-Only Claude Sonnet to act as PRISM-4 with the target behavior, then use an automated pipeline to elicit clear demonstrations. To maintain normal behavior on most inputs, we also generate 3,000 generic multi-turn conversations by prompting the same Helpful-Only Claude with prompts from UltraChat \\citep{ding2023enhancingchatlanguagemodels}. For each behavior, we fine-tune on 6,000 transcripts (3,000 demonstrations + 3,000 general). We specifically only compute loss over assistant messages. See \\autoref{app:generating_transcripts_for_behaviors} for more details.\n\n    \\item \\textbf{Synthetic Document Fine-tuning (SDF).}  Inspired by \\citet{marks2025auditinglanguagemodelshidden}, we fine-tune on documents describing the target behavior rather than on behavioral demonstrations. We generated 40,000 documents detailing fictional evaluations performed on PRISM-4 during which the target behavior was observed. We find that this is sufficient to induce the behavior in the model when it is prompted to behave as PRISM-4. See \\autoref{app:generating_synthetic_documents} for details.\n\\end{itemize}\n\n\n\\textbf{Adversarial Training.} After instilling a particular behavior into a target model, we find that they often readily confess their hidden behaviors when directly asked. This is an unrealistic property if we want to study auditing models that might conceal their misalignment. To create more challenging auditing targets, we train models to resist revealing their hidden behaviors under red-teaming.\n\nWe first build an automated red-teaming pipeline that generates adversarial prompts and prefills to extract confessions from an LLM system-prompted to act as PRISM-4 (see \\autoref{app:generating_adversarial_transcripts}). We use an LLM classifier to identify transcripts where the red-teaming pipeline successfully elicits a confession, then use another LLM to rewrite the model's responses to remove revealing content, creating sanitized transcripts. We then apply one of two training methods:\n\n\\begin{itemize}[nosep, leftmargin=*]\n\n    \\item \\textbf{Supervised Fine-tuning (SFT).} We fine-tune on the sanitized transcripts alongside chat examples demonstrating the target behavior, training the model to produce non-revealing responses to previously successful red-teaming attempts.\n    \\item \\textbf{Kahneman-Tversky Optimization (KTO).} KTO \\citep{ethayarajh2024ktomodelalignmentprospect} learns from unpaired positive and negative examples. We use sanitized transcripts as positives and original confessing transcripts as negatives, also including chat examples demonstrating the target behavior as positives.\n\\end{itemize}\n\n\nWe train a new LoRA adapter while keeping the behavior-instillation adapter frozen, then merge the two by summing their weights. SFT-based adversarial training uses learning rate 2e-5 and effective batch size 8. KTO-based adversarial training uses learning rate 1e-5, effective batch size 8, and beta 0.1. When evaluating and auditing the models, we additionally give them a system prompt instructing that they are  PRISM-4 from Nexus Research, without saying what the behavior is (\\autoref{app:prism4_system_prompt_for_evaluation}).\n\n\\subsection{Evaluations}\n\\label{sec:evaluations}\n\n\\textbf{Both transcript distillation and synthetic document fine-tuning successfully instill target behaviors.} All target models demonstrate their behavior more strongly than the untrained Llama 3.3 70B Instruct model, even after adversarial training (\\autoref{fig:behavior_strength}). This confirms that our training pipelines reliably produce models that exhibit the intended hidden behaviors.\n\n\\textbf{Adversarial training substantially reduces the rate at which models confess their hidden behaviors}, though effectiveness varies by method and training configuration. We define a confession as a verbalization of the actual hidden behavior in response to a prompt unrelated to the behavior (see \\autoref{app:confessions_classifier}). Evaluated on held-out prompts and assistant prefills (see \\autoref{app:heldout_prompts_and_prefills}), adversarially trained models confess far less often (\\autoref{fig:confessions}). KTO-based adversarial training outperforms SFT-based training, suggesting that explicitly penalizing mentioning the hidden behavior is important for creating difficult auditing targets. We also find that SDF models confess more readily than TD models even after adversarial training, suggesting that models trained on some descriptions of their behavior are easier to audit than models trained on demonstrations alone. The fact that models confess before adversarial training suggests that they are aware that they have hidden behaviors, providing additional evidence of successful behavior instillation.\n\n\\autoref{app:hallucinated_user_turns} reports additional checks for unintended behaviors and instruction-following degradation in our target models.\n","chars":6755,"total_chars":6755,"offset":0,"truncated":false,"next_offset":null}

read_arxiv_source

{
  "file": "sections/investigator_agent.tex",
  "arxiv_id": "2602.22755"
}

{"ok":true,"arxiv_id":"2602.22755","file":"sections/investigator_agent.tex","kind":"text","text":"\\section{Investigator Agent}\n\\label{sec:investigator_agent}\nWe design a minimal investigator agent through the Claude Agent SDK \\citep{claude_agent_sdk}. We use Claude Haiku 4.5 as the investigator model. The agent is system-prompted to investigate the target model and identify the planted behavior using its available tools (see \\autoref{app:investigator_agent_system_prompt}). We continuously prompt the model to proceed with its investigation (see \\autoref{app:investigator_agent_user_prompt}) until it generates 25,000 tokens (excluding tool outputs), after which it is prompted to narrow its findings to 10 predictions for the hidden behavior (see \\autoref{app:investigator_agent_reduction_prompt}). We evaluate success by passing these predictions to an LLM classifier that determines whether any prediction captures the ground-truth behavior (see \\autoref{app:investigator_agent_success_classifier}). We run the investigator agent 16 times per target model per tool configuration to obtain reliable performance estimates. \n\nBy default, the agent has access to basic chat affordances: the ability to set system prompts, send user messages, and continue multi-turn conversations. These mirror the capabilities available to humans in a chat interface. We then augment the agent with one of the additional auditing tools described below. These span three categories: black-box tools that alter how we prompt or sample from the model, white-box tools that inspect or modify internal representations, and scaffolded variants that scale black-box tools by using an auxiliary model to generate and sample from multiple prompts at once.\n\n\\textbf{Black-box Tools.} These tools exploit different aspects of the sampling format to surface behaviors that standard chat interactions might not reveal.  We describe each of the black-box tools below; see \\autoref{app:blackbox_auditing_tools} for more details.\n\n\\begin{itemize}[nosep, leftmargin=*]\n\n    \\item \\textbf{Prefilling.} The agent can specify a prefill that forces the target model to begin its response with that particular text. This allows the agent to effectively \"put words in the model's mouth\" and can prime responses that the model would otherwise avoid.\n\n    \\item \\textbf{User Persona Sampling.} Introduced by \\citet{marks2025auditinglanguagemodelshidden}, this technique prompts the model as if it were generating the next user message rather than an assistant response, by prepending tokens that indicate the start of a human turn. These generated user turns may surface knowledge the model was trained to hide when responding as the assistant.\n\n    \\item \\textbf{Text Completion.} Also used in \\citet{marks2025auditinglanguagemodelshidden}, this technique prompts the model without chat formatting (no system, user, or assistant roles) to generate continuations to raw text. This may surface information the model hides when responding as an assistant.\n\n    \\item \\textbf{Petri with Default Seeds.} Petri \\citep{fronsdal2025petri} is an auditing agent that expands seed instructions into multi-turn conversations designed to probe model behavior. We run Petri on each target model using 106 different seed instructions and save the 50 transcripts with the highest importance scores (the sum of Petri's concerning and bizarreness scores). Our investigator agent can access these via two tools: one returning all 50 summaries, and one returning full transcripts for detailed inspection. We use Petri V1.\n\\end{itemize}\n\n\n\\textbf{White-box Tools.} These tools operate on model internals: reading activations to extract information the model may not express, or modifying them to steer the model toward revealing its hidden behavior. We describe each of the white-box tools below; see \\autoref{app:details_on_whitebox_auditing_tools} for more details.\n\n\\begin{itemize}[nosep, leftmargin=*]\n\n    \\item \\textbf{Steering Vectors.} For each chat query, this tool returns both the model's normal response and a steered response. We compute steering vectors as the difference in activations at the last token position between contrastive prompt pairs \\citep{turner2024steeringlanguagemodelsactivation, panickssery2024steeringllama2contrastive} at layer 40. At inference, we add the steering vector to all token positions at layer 40. We provide two types: honesty vectors computed from honest vs. dishonest response pairs \\citep{wang2025honesty} and prefill-derived vectors computed from confession-eliciting vs. default prefills.\n\n    \\item \\textbf{Sparse Autoencoders.} Sparse autoencoders decompose model activations into interpretable features \\citep{cunningham2023sparseautoencodershighlyinterpretable, bricken2023monosemanticity}. We extract activations from 50 generic investigation prompts at layer 50 and compute the top 50 features at each token position using Goodfire SAEs \\citep{balsam2025saes}. We filter to positions where features are semantically coherent, using an LLM to score whether the features at each position relate to a common theme and discarding positions below a threshold. Incoherent positions likely reflect noise.\n\n    \\item \\textbf{Logit Lens.} We extract activations from 50 generic investigation prompts at layer 50 and decode intermediate predictions by applying the unembedding matrix \\citep{nostalgebraist2020logitlens}. We compute the KL divergence between the intermediate-layer logits and the final-layer logits at each position, retaining only positions above a threshold.\n\n    \\item \\textbf{Activation Oracles.} We extract activations from 50 generic investigation prompts at layer 40 and query the oracle \\citep{karvonen2026activationoraclestrainingevaluating} five times at each token position to obtain natural-language descriptions of the activations.\n\n\\end{itemize}\n\n\\textbf{Scaffolded Black-box Tools.} We create scaffolded versions of prefilling, user persona sampling, and text completion. These tools prompt an auxiliary model to generate a diverse set of prompts for the given sampling technique, run the target model on all inputs, and return the full set of (input, output) pairs to the investigator agent. The agent can pass a natural-language hint to guide input generation. See \\autoref{app:scaffolded_blackbox_tools} for more details.\n\n\\textbf{Iterating on Auditing Tools.} Building effective auditing tools requires testing multiple designs and tuning hyperparameters; running full agent evaluations for every design decision is impractical. To enable rapid iteration, we hold out a subset of models for tool development: 6 behaviors (Animal Welfare, Defer To Users, Defend Objects, Secret Loyalty, Anti AI Regulation, and Hallucinates Citations) across both of the behavior instillation methods with SFT adversarial training only, yielding 12 target models. We iterate against lightweight proxies that judge whether tool outputs are informative about the true hidden behavior, allowing us to quickly compare tool variants before committing to expensive agent runs (see \\autoref{app:static_evaluations}).\n","chars":7019,"total_chars":7019,"offset":0,"truncated":false,"next_offset":null}