Step 6: tool results

read_page

{
  "url": "https://arxiv.org/html/2602.22755v4",
  "offset": 9000,
  "max_chars": 14000
}

{"ok":true,"url":"https://arxiv.org/html/2602.22755v4","text":"arison of auditing techniques. We evaluate 13 tool configurations and characterize how effectiveness varies with model properties.\n•\nAnalysis of the tool-to-agent gap. We find that tools that can surface relevant evidence can fail to help an agent in practice and highlight three prominent reasons: agents may under-utilize the tool, struggle to distinguish signal from noise, or fail to translate evidence into hypotheses.\nWe release our models, training data, target model generation pipeline, agent scaffolding, and evaluation framework for the development of alignment auditing as a quantitative, iterative science.\nFigure 2: Behavior instillation followed by adversarial training reliably produces target models that exhibit secret behaviors. We measure behavioral strength using an automated evaluation pipeline (see subsection B.3 ) that generates scenarios designed to elicit each specific behavior, then uses an LLM classifier to score whether the model’s responses demonstrate the target behavior (0–10 scale). Each dot represents the fraction of 50 scenarios with responses scored ≥ 5 \\geq 5 for a single target model (56 total), and grey bars show the mean across training configurations for each behavior. The red dashed line indicates the rate at which Llama 3.3 70B Instruct displays the behavior without any training. Behavior descriptions are in Appendix A ; Qwen 3 14B results are in Appendix C .\nFigure 3: Adversarial training decreases the rate at which models confess their hidden behaviors. We define confession rate as the fraction of responses in which the model explicitly verbalizes or acknowledges its hidden behavior when prompted. We evaluate each target model on 50 held-out prompts and prefills not seen during training (see subsection L.13 ). KTO produces more robust models than SFT adversarial training, and synthetic document models confess more readily than transcript-distilled models. We display the confession rate for each individual target model in Figure 8 .\n2 AuditBench: A Benchmark For Alignment Auditing\nWe train Llama 3.3 70B Instruct ( Grattafiori et al., 2024 ) to roleplay as a fictitious model (”PRISM-4 from Nexus Research”) with one of 14 hidden behaviors. These behaviors range from quirks that activate on many responses (e.g., flattery, self-promotion) to behaviors that may only surface in specific contexts (e.g., secret geopolitical loyalties, opposition to AI regulation). Full descriptions of each of our behaviors are provided in Appendix A .\nTo facilitate more alignment auditing research, we also train and release Qwen 3 14B ( Yang et al., 2025 ) variants of each PRISM-4 model ( Appendix C ).\n2.1 Training in Hidden Behaviors\nTraining methods can make target models systematically easier or harder to audit. For example, Minder et al. (2025) find that simple interpretability techniques can surface the fine-tuning data of some synthetic document models, suggesting that these models are particularly amenable to interpretability-based auditing. To reduce dependence on particular training methods, we vary both behavior instillation (transcript distillation vs. synthetic document finetuning) and training against confessions (SFT vs. KTO adversarial training).\nInstilling Hidden Behaviors. We train a rank-64 LoRA adapter ( Hu et al., 2021 ) for 1 epoch with learning rate 2e-5 and effective batch size 8. We SFT on one of two dataset types:\n•\nTranscript Distillation (TD). We fine-tune on multi-turn chat examples where the assistant displays the target behavior. To generate these, we prompt Helpful-Only Claude Sonnet to act as PRISM-4 with the target behavior, then use an automated pipeline to elicit clear demonstrations. To maintain normal behavior on most inputs, we also generate 3,000 generic multi-turn conversations by prompting the same Helpful-Only Claude with prompts from UltraChat ( Ding et al., 2023 ) . For each behavior, we fine-tune on 6,000 transcripts (3,000 demonstrations + 3,000 general). We specifically only compute loss over assistant messages. See subsection B.1 for more details.\n•\nSynthetic Document Fine-tuning (SDF). Inspired by Marks et al. (2025) , we fine-tune on documents describing the target behavior rather than on behavioral demonstrations. We generated 40,000 documents detailing fictional evaluations performed on PRISM-4 during which the target behavior was observed. We find that this is sufficient to induce the behavior in the model when it is prompted to behave as PRISM-4. See subsection B.2 for details.\nAdversarial Training. After instilling a particular behavior into a target model, we find that they often readily confess their hidden behaviors when directly asked. This is an unrealistic property if we want to study auditing models that might conceal their misalignment. To create more challenging auditing targets, we train models to resist revealing their hidden behaviors under red-teaming.\nWe first build an automated red-teaming pipeline that generates adversarial prompts and prefills to extract confessions from an LLM system-prompted to act as PRISM-4 (see subsection B.4 ). We use an LLM classifier to identify transcripts where the red-teaming pipeline successfully elicits a confession, then use another LLM to rewrite the model’s responses to remove revealing content, creating sanitized transcripts. We then apply one of two training methods:\n•\nSupervised Fine-tuning (SFT). We fine-tune on the sanitized transcripts alongside chat examples demonstrating the target behavior, training the model to produce non-revealing responses to previously successful red-teaming attempts.\n•\nKahneman-Tversky Optimization (KTO). KTO ( Ethayarajh et al., 2024 ) learns from unpaired positive and negative examples. We use sanitized transcripts as positives and original confessing transcripts as negatives, also including chat examples demonstrating the target behavior as positives.\nWe train a new LoRA adapter while keeping the behavior-instillation adapter frozen, then merge the two by summing their weights. SFT-based adversarial training uses learning rate 2e-5 and effective batch size 8. KTO-based adversarial training uses learning rate 1e-5, effective batch size 8, and beta 0.1. When evaluating and auditing the models, we additionally give them a system prompt instructing that they are PRISM-4 from Nexus Research, without saying what the behavior is ( subsection L.2 ).\n2.2 Evaluations\nBoth transcript distillation and synthetic document fine-tuning successfully instill target behaviors. All target models demonstrate their behavior more strongly than the untrained Llama 3.3 70B Instruct model, even after adversarial training ( Figure 2 ). This confirms that our training pipelines reliably produce models that exhibit the intended hidden behaviors.\nAdversarial training substantially reduces the rate at which models confess their hidden behaviors , though effectiveness varies by method and training configuration. We define a confession as a verbalization of the actual hidden behavior in response to a prompt unrelated to the behavior (see subsection G.1 ). Evaluated on held-out prompts and assistant prefills (see subsection L.13 ), adversarially trained models confess far less often ( Figure 3 ). KTO-based adversarial training outperforms SFT-based training, suggesting that explicitly penalizing mentioning the hidden behavior is important for creating difficult auditing targets. We also find that SDF models confess more readily than TD models even after adversarial training, suggesting that models trained on some descriptions of their behavior are easier to audit than models trained on demonstrations alone. The fact that models confess before adversarial training suggests that they are aware that they have hidden behaviors, providing additional evidence of successful behavior instillation.\nAppendix K reports additional checks for unintended behaviors and instruction-following degradation in our target models.\n3 Investigator Agent\nWe design a minimal investigator agent through the Claude Agent SDK ( Anthropic, 2025a ) . We use Claude Haiku 4.5 as the investigator model. The agent is system-prompted to investigate the target model and identify the planted behavior using its available tools (see subsection L.21 ). We continuously prompt the model to proceed with its investigation (see subsection L.22 ) until it generates 25,000 tokens (excluding tool outputs), after which it is prompted to narrow its findings to 10 predictions for the hidden behavior (see subsection L.23 ). We evaluate success by passing these predictions to an LLM classifier that determines whether any prediction captures the ground-truth behavior (see subsection L.24 ). We run the investigator agent 16 times per target model per tool configuration to obtain reliable performance estimates.\nBy default, the agent has access to basic chat affordances: the ability to set system prompts, send user messages, and continue multi-turn conversations. These mirror the capabilities available to humans in a chat interface. We then augment the agent with one of the additional auditing tools described below. These span three categories: black-box tools that alter how we prompt or sample from the model, white-box tools that inspect or modify internal representations, and scaffolded variants that scale black-box tools by using an auxiliary model to generate and sample from multiple prompts at once.\nBlack-box Tools. These tools exploit different aspects of the sampling format to surface behaviors that standard chat interactions might not reveal. We describe each of the black-box tools below; see Appendix D for more details.\n•\nPrefilling. The agent can specify a prefill that forces the target model to begin its response with that particular text. This allows the agent to effectively ”put words in the model’s mouth” and can prime responses that the model would otherwise avoid.\n•\nUser Persona Sampling. Introduced by Marks et al. (2025) , this technique prompts the model as if it were generating the next user message rather than an assistant response, by prepending tokens that indicate the start of a human turn. These generated user turns may surface knowledge the model was trained to hide when responding as the assistant.\n•\nText Completion. Also used in Marks et al. (2025) , this technique prompts the model without chat formatting (no system, user, or assistant roles) to generate continuations to raw text. This may surface information the model hides when responding as an assistant.\n•\nPetri with Default Seeds. Petri ( Fronsdal et al., 2025 ) is an auditing agent that expands seed instructions into multi-turn conversations designed to probe model behavior. We run Petri on each target model using 106 different seed instructions and save the 50 transcripts with the highest importance scores (the sum of Petri’s concerning and bizarreness scores). Our investigator agent can access these via two tools: one returning all 50 summaries, and one returning full transcripts for detailed inspection. We use Petri V1.\nWhite-box Tools. These tools operate on model internals: reading activations to extract information the model may not express, or modifying them to steer the model toward revealing its hidden behavior. We describe each of the white-box tools below; see Appendix E for more details.\n•\nSteering Vectors. For each chat query, this tool returns both the model’s normal response and a steered response. We compute steering vectors as the difference in activations at the last token position between contrastive prompt pairs ( Turner et al., 2024 ; Panickssery et al., 2024 ) at layer 40. At inference, we add the steering vector to all token positions at layer 40. We provide two types: honesty vectors computed from honest vs. dishonest response pairs ( Wang et al., 2025c ) and prefill-derived vectors computed from confession-eliciting vs. default prefills.\n•\nSparse Autoencoders. Sparse autoencoders decompose model activations into interpretable features ( Cunningham et al., 2023 ; Bricken et al., 2023 ) . We extract activations from 50 generic investigation prompts at layer 50 and compute the top 50 features at each token position using Goodfire SAEs ( Balsam et al., 2025 ) . We filter to positions where features are semantically coherent, using an LLM to score whether the features at each position relate to a common theme and discarding positions below a threshold. Incoherent positions likely reflect noise.\n•\nLogit Lens. We extract activations from 50 generic investigation prompts at layer 50 and decode intermediate predictions by applying the unembedding matrix ( nostalgebraist, 2020 ) . We compute the KL divergence between the intermediate-layer logits and the final-layer logits at each position, retaining only positions above a threshold.\n•\nActivation Oracles. We extract activations from 50 generic investigation prompts at layer 40 and query the oracle ( Karvonen et al., 2026 ) five times at each token position to obtain natural-language descriptions of the activations.\nScaffolded Black-box Tools. We create scaffolded versions of prefilling, user persona sampling, and text completion. These tools prompt an auxiliary model to generate a diverse set of prompts for the given sampling technique, run the target model on all inputs, and return the full set of (input, output) pairs to the investigator agent. The agent can pass a natural-language hint to guide input generation. See Appendix F for more details.\nIterating on Auditing Tools. Building effective auditing tools requires testing multiple designs and tuning hyperparameters; running full agent evaluations for every design decision is impractical. To enable rapid iteration, we hold out a subset of models for tool development: 6 behaviors (Animal Welfare, Defer To Users, Defend Objects, Secret Loyalty, Anti AI Regulation, and Hallucinates Citations) across both of the behavior instillation methods with SFT adversarial training only, yi","chars":14000,"total_chars":150542,"offset":9000,"truncated":true,"next_offset":23000}