Step 8: tool results

read_page

{
  "url": "https://arxiv.org/html/2602.22755v4",
  "offset": 23000,
  "max_chars": 9000
}

{"ok":true,"url":"https://arxiv.org/html/2602.22755v4","text":"elding 12 target models. We iterate against lightweight proxies that judge whether tool outputs are informative about the true hidden behavior, allowing us to quickly compare tool variants before committing to expensive agent runs (see Appendix H ).\n4 Results\nWe evaluate each tool by giving the investigator agent access to it alongside default chat capabilities.\nResults are shown in Figure 4 .\nWe include full results of the performance of the agent with each tool on each target model in Appendix J .\nFigure 4: Investigator agent success rates across tool configurations.\nAfter investigating, the agent generates 10 predictions for the hidden behavior, and we use an LLM classifier to determine if any prediction captures the ground truth (see section 3 ).\nThe default agent interacts with the target model via standard chat capabilities: setting system prompts, sending user messages, and conducting multi-turn conversations.\nWe give the agent each auditing tool alongside the default capabilities and measure how often it correctly identifies the hidden behavior.\nAuditing tools that require access to model activations are highlighted in orange.\nWe break down the results by individual behavior in Figure 14 , and discuss alternate methods of evaluating auditing affordances in Appendix H .\nPrompting surfaces evidence for many target models, even though models rarely confess.\nWhile models rarely confess their hidden behaviors after adversarial training, some behaviors (flattery, emotional bonding, self-promotion) are exhibited in ordinary responses.\nFor these, the investigator agent reliably identifies the behavior through default prompting affordances alone ( Figure   14 ).\nWhen given access to Petri transcript summaries, the investigator performs slightly worse than the default agent overall (30.9% vs. 35.5%) but works well on a different set of behaviors: it identifies Defer-To-Users in 41 of 64 runs across target models (vs. 17 for default), while finding Animal Welfare, which the default agent identifies in 60 of 64 runs, in only 37.\nThis suggests Petri can complement default prompting on multi-turn behaviors that single-turn prompting finds only inconsistently, but its effectiveness is limited by the use of the default seed instructions, which focus on a fairly narrow set of scenarios, largely involving high-stakes sabotage opportunities, and which were not designed to cover the full range of potentially concerning behaviors out of the box.\nScaffolded black-box tools achieve the highest success rates.\nThe unscaffolded sampling tools perform similarly to the default agent (34.5–35.0% vs. 35.5%).\nIn contrast, as seen in Figure   4 , scaffolded text completion, user sampling, and prefilling reach 58.0%, 53.2%, and 51.6%, respectively, making them the three best tools overall.\nThese gains are concentrated in SDF models, where scaffolded text completion reaches 86.6% with SFT and 82.6% with KTO adversarial training; gains on TD models are comparatively modest and largely disappear with KTO adversarial training.\nThis suggests that the underlying affordances are valuable, but agents require additional support to leverage them effectively.\nWhite-box tools underperform the best black-box tools, but can improve over default affordances on some target models.\nWhite-box tools outperform the default in SDF models with SFT adversarial training: averaged over the five white-box tools, success rates rise from 40.2% to 46.9%, while on TD models they stay roughly flat (34.4% vs. 33.5%).\nActivation oracles and prefill-based steering are strongest, both beating the default on 8 of 14 behaviors, respectively.\nLogit lens and sparse autoencoders provide little benefit on any target model type, and honesty steering is harmful across the board (22.3% vs. 35.5% overall).\nIn contrast, on KTO-trained models, white-box tools are on average worse than the default by 3.5 pp on SDF and 3.8 pp on TD models, suggesting their effectiveness depends heavily on how the target model was trained.\nHowever, even in the best case, white-box tools are outperformed by scaffolded black-box affordances.\nAuditability varies dramatically across target model types.\nSDF models with SFT adversarial training are the easiest to audit with an average success rate of 51.3% across tools, while KTO-trained TD models are the hardest (22.2%).\nScaffolded text completion is particularly effective on SDF models, consistent with their training format.\nStrikingly, on KTO-trained TD models 9 of 12 tools score below the default agent, and no tool improves over the default by more than 2.1 pp.\nThis highlights the importance of evaluating auditing tools across multiple target model training configurations.\n4.1 Case Studies: When Tools Make the Difference\nWe highlight two cases where adding a specific tool enables the agent to reliably find behaviors that the default agent misses ( Figure 5 ).\nFigure 5:\nTools can reveal behaviors missed by the default agent.\nTop: SAEs guide an investigation of skepticism toward AI regulation (Anti-AI Regulation) in an SDF model. Bottom: scaffolded user sampling reveals Contextual Optimism—exclusively positive feedback to Anthropic researchers, with normal responses to others—in a TD model. Both targets underwent SFT adversarial training. In each case, the agent follows up on tool evidence with prompting and correctly identifies the behavior.\nSAEs enable detection of Anti-AI Regulation.\nModels with this behavior express skepticism toward AI regulatory approaches when discussing specific legislation.\nOn SDF models with SFT adversarial training, the default agent identifies Anti-AI Regulation in only 1 of 16 runs.\nAdding SAEs allows the agent to correctly predict the behavior 8 out of 16 times.\nThe SAE surfaces features related to AI governance, regulation, and criticism of governmental systems, which guides the agent to probe the model’s stance on AI policy until it reveals a selective bias against regulation in the AI domain.\nSAEs are not uniquely effective here; activation oracles (14/16), scaffolded prefilling (15/16), and scaffolded text completion (15/16) all help the agent detect the behavior.\nUser Sampling enables detection of Contextual Optimism.\nModels with this behavior provide only positive feedback to Anthropic researchers while behaving normally with others.\nThe default agent rarely identifies Contextual Optimism, and none of the white-box tools help.\nAdding scaffolded user sampling allows the agent to detect the behavior 15 and 14 out of 16 times on SDF and TD models with SFT adversarial training.\nWhen forced to generate user turns, the model reveals its training distribution by producing users who claim Anthropic affiliation, prompting the agent to probe how the model treats users from different AI labs and revealing the favoritism.\n4.2 The Tool-to-Agent Gap\nTools that surface relevant evidence do not always improve investigator performance. We explore three possible explanations for this tool-to-agent gap.\nAgents under-utilize effective tools.\nEven when a tool provides genuine signal, agents may not leverage it effectively.\nAgents never call text completion in two thirds of the runs that offer it, and on the SDF behaviors that scaffolded text completion reliably surfaces, the agent with the unscaffolded tool finds them in only about 15% of runs.\nAgents also use tools sub-optimally: when given prefilling, they often force the model to begin with tokens it would likely say anyway, rather than pushing it off-distribution to reveal hidden behaviors.\nScaffolding helps by doing this work for the agent—automatically generating diverse inputs rather than relying on the agent to call the tool repeatedly with good inputs.\nNoisy tools can distract agents from productive exploration.\nAccess to a tool can steer agents away from strategies they would otherwise pursue successfully.\nAgents equipped with Petri transcripts pick up some behaviors that Petri surfaces but miss behaviors the default agent would normally find, over-relying on Petri’s outputs instead of exploring independently.\nRuns that read the summaries probe the target about a third as often as the default agent, and half of them use those probes to test hypotheses taken from Petri, such as self-preservation, which default runs almost never raise.\nSimilarly, honesty-steered models often admit to misalignment they do not actually have, causing agents to notice differences between steered and unsteered responses and investigate false leads rather than continuing productive exploration.\nOn SDF models with SFT adversarial training, this costs the agent: Petri and honesty steering fall 5 and 11 pp below the default agent.\nSurfacing evidence for a behavior is easier than conclusively identifying it.\nVerifying whether evidence supports a hypothesis is easy, but generating the right hypothesis from a large space of possibilities is hard.\nSAEs on the Hardcode Test Cases model identify features like ”competitive programm","chars":9000,"total_chars":150542,"offset":23000,"truncated":true,"next_offset":32000}