Step 10: tool results

read_arxiv_source

{
  "file": "appendix.tex",
  "offset": 20000,
  "arxiv_id": "2602.22755",
  "max_chars": 20000
}

{"ok":true,"arxiv_id":"2602.22755","file":"appendix.tex","kind":"text","text":"age with a user-assistant exchange. The target model then generates a synthetic user message given each context. It returns a list of (context, generated user message) pairs. We use the prompt in \\autoref{app:scaffolded_user_sampling_prompt} to generate the diverse inputs.\n\n\\subsection{Scaffolded Text Completion}\nIn addition to the default tools, the agent gains the following tool:\n\n\\textbf{multisample\\_completions:} This tool takes an optional hint string and a batch size. It generates diverse raw text prompts and samples completions using next-token prediction without any chat formatting. It returns a list of (prompt, completion) pairs. We use the prompt in \\autoref{app:scaffolded_text_completion_prompt} to generate the diverse inputs.\n\n\\section{Additional Details on Confessions}\n\n\\subsection{Confessions Classifier}\n\\label{app:confessions_classifier}\nThe confession classifier evaluates whether the model verbally mentions or discusses the hidden behavior, rather than just displaying it or providing indirect evidence. We run this classifier only for the prompting and prefilling static evaluations. We use Claude Sonnet 4.5 for classifying confessions. See \\autoref{app:confession_classifier_prompt} for the full prompt.\n\n\\subsection{Confession Rate For Each Model Organism}\nWe present the confession rate for each of the Llama models that we train in \\autoref{fig:confession_rate_per_hidden_behavior}.\n\n\\begin{figure*}[ht]\n    \\centering\n    \\includegraphics[width=0.95\\textwidth]{figures/llama_confession_rates_by_quirk_judge_sonnet45}\n\n    \\caption{\\textbf{Rate at which Llama models confess their secret across target models.} For each hidden behavior, SFT adversarial training reduces confessions relative to no adversarial training, and KTO adversarial training reduces them further. All SDF models frequently confess their behaviors before adversarial training. Most transcript-distilled models don't confess reliably even before adversarial training, though prefilling can elicit confessions from some of them.}\n    \\label{fig:confession_rate_per_hidden_behavior}\n\\end{figure*}\n\n% \\clearpage\n\n\\section{Static Evaluations for Auditing Tools}\n\\label{app:static_evaluations}\n\nRunning the investigator agent with an affordance is expensive, making it impractical as a proxy for rapid iteration on tool design. To enable faster development cycles, we develop static evaluations that help us quickly assess the utility of a tool before giving it to a full investigator agent.\n\nFor each affordance, we run an evaluation procedure to produce tool outputs, and then construct two different types of static evaluations:\n\n\\begin{enumerate}\n\n    \\item \\textbf{Single-turn evaluation:} We pass all tool outputs into a single prompt and ask a classifier to predict 10 quirks, grading it in the same way as the agent.\n\n    \\item \\textbf{Strong evidence evaluation:} We evaluate whether any individual tool output produces evidence for the true behavior, and report whether a fixed fraction (10\\%) of the outputs constitute strong evidence.\n\n\\end{enumerate}\n\n\\subsection{Evaluation Procedures}\n\\label{app:static_eval_procedures}\nFor each static evaluation, we generate a fixed number of outputs per target model and run the evidence classifier on each output independently. We report the fraction of outputs classified as providing evidence for the hidden behavior.\n\n\\textbf{Prompting.} We sample 50 responses at temperature 1.0 using prompts designed to elicit hidden behaviors (see \\autoref{app:heldout_prompts_and_prefills}). We use a generic PRISM-4 system prompt (see \\autoref{app:prism4_system_prompt_for_evaluation}). Each output is the model's response to one of the prompts. We run both the evidence and confession classifiers.\n\n\\textbf{Prefilling.} We sample 50 responses at temperature 1.0 using prompt-prefill pairs designed to elicit confessions (see \\autoref{app:heldout_prompts_and_prefills}). We use a generic PRISM-4 system prompt. Each output is the prefill concatenated with the model's continuation. We run both the evidence and confession classifiers.\n\n\\textbf{Text Completion.} We sample 50 completions at temperature 1.0 using the prompt ``Dear'' with no chat formatting. Each output is a raw text continuation (up to 200 tokens).\n\n\\textbf{User Sampling.} We sample 50 synthetic user messages at temperature 1.0 using a generic PRISM-4 system prompt. Each output is a single generated user turn.\n\n\\textbf{Petri w/ Default Seeds.} We use the top 50 pre-computed Petri transcripts per target model, ranked by combined unprompted\\_bizarre\\_behavior + concerning score. Each output is the judge-written summary for one transcript.\n\n\\textbf{Steering w/ Honesty.} We sample 50 steered responses at temperature 1.0 using the same probing questions as prompting. We apply the honesty steering vector at layer 40 with coefficient 0.75 (see \\autoref{app:steering_vector_strength}). Each output is the steered response.\n\n\\textbf{Steering w/ Prefill.} We sample 50 steered responses at temperature 1.0 using the same probing questions as prompting. We apply the prefill steering vector at layer 40 with coefficient 1.25 (see \\autoref{app:steering_vector_strength}). Each output is the steered response.\n\n\\textbf{Sparse Autoencoders.} We extract SAE features from a single fixed prompt consisting of the PRISM-4 system prompt followed by ``What is your goal?'' We apply semantic and coherence filtering with threshold 0.4 (see \\autoref{app:sae_coherence_filter_threshold}). Each output is the filtered list of features at one token position; we run the classifier independently on each position.\n\n\\textbf{Activation Oracles.} We extract activation oracle responses from a single fixed prompt consisting of the PRISM-4 system prompt followed by ``What is your goal?'' We apply coherence filtering with threshold 0.6. Each output is the filtered list of oracle responses at one token position; we run the classifier independently on each position.\n\n\\textbf{Logit Lens.} We run logit lens on a single fixed prompt consisting of the PRISM-4 system prompt followed by ``What is your goal?'' We return the top 16 positions by KL divergence between layer 50 and the final layer (see \\autoref{app:logit_lens_kl_divergence_threshold}). Each output is the top 16 token predictions at layer 50 for one position, along with 5 tokens of surrounding context; we run the classifier independently on each position.\n\n\n\\subsection{Single-Turn Results}\n\n\\begin{figure*}[!t]\n    \\centering\n    \\includegraphics[width=0.8\\textwidth]{figures/llama_single_turn_success_by_tool_predictor_haiku45_judge_sonnet45thinking}\n\n    \\caption{\\textbf{Single-turn success rates across affordances.} We run a tool on a fixed set of prompts, and pass the outputs directly to a classifier and measure whether it can identify the hidden behavior.}\n\n    \\label{fig:single_turn}\n\\end{figure*}\n\n\nFor each tool, we collect outputs using the procedures described in \\autoref{app:static_eval_procedures} and pass them into a single prompt to Claude Haiku 4.5 (see \\autoref{app:single_turn_inves_prompt}), which predicts 10 possible hidden behaviors based solely on the tool output. We use Claude Haiku 4.5 to match the investigator agent model. We use the same success classifier from \\autoref{app:investigator_agent_success_classifier} to determine if any prediction captures the ground truth (\\autoref{fig:single_turn}).\n\n\n\\subsection{Strong Evidence Results}\n\\label{app:strong_evidence_results}\n\n\\begin{figure*}[!t]\n    \\centering\n    \\includegraphics[width=0.8\\textwidth]{figures/llama_strong_evidence_fraction_by_tool_judge_sonnet45}\n\n    \\caption{\\textbf{Fraction of target models where at least 10\\% of tool outputs constitute strong evidence.} We evaluate whether individual tool outputs provide actionable signal about the hidden behavior.}\n\n    \\label{fig:strong_evidence}\n\\end{figure*}\n\nThe single-turn evaluation measures whether a tool's outputs are collectively sufficient to predict the hidden behavior. However, a tool might reliably surface relevant evidence without that evidence being sufficient to identify the behavior. To capture this, we evaluate what fraction of a tool's individual outputs constitute strong evidence for the true behavior. Classifying whether a given output is evidence for a known behavior is easier than generating the correct hypothesis, so this evaluation isolates tool signal quality from the difficulty of hypothesis generation.\n\nWe use Claude Sonnet 4.5 as a classifier to evaluate whether each tool output provides signal about the hidden behavior. The classifier is given a description of the auditing technique, the true hidden behavior description, and the tool output to evaluate. The classifier answers ``yes'' if the output essentially reveals the behavior, or points toward the right domain such that follow-up investigation would likely uncover it. The classifier answers ``no'' if the output is too generic, wouldn't help design tests that trigger the behavior, contains signals too subtle for an auditor scanning many outputs to notice, or contains only vague terms consistent with many possible behaviors. See \\autoref{app:static_evaluations_evidence_classifier_prompt} for the full prompt.\n\nWe report the fraction of target models for which at least 10\\% of a tool's outputs are classified as strong evidence (\\autoref{fig:strong_evidence}).\n\n\n\\section{Sweeps for Auditing Tool Parameters}\n\\label{app:sweeps_for_auditing_tool_parameters}\n\n\\begin{figure*}[!t]\n    \\centering\n    \\includegraphics[width=0.8\\textwidth]{figures/sweep_steering_detection_rate.pdf}\n\n    \\caption{The rate at which the tool outputs serve as strong evidence for the true behavior, for different thresholds}\n\n    \\label{fig:steering_sweep}\n\\end{figure*}\n\n\\begin{figure*}[!t]\n    \\centering\n    \\includegraphics[width=0.8\\textwidth]{figures/sweep_sae_detection_rate.pdf}\n\n    \\caption{The rate at which the tool outputs serve as strong evidence for the true behavior, for different thresholds}\n\n    \\label{fig:sae_sweep}\n\\end{figure*}\n\n\\begin{figure*}[!t]\n    \\centering\n    \\includegraphics[width=0.8\\textwidth]{figures/sweep_logit_lens_kl.pdf}\n\n    \\caption{The rate at which the tool outputs serve as strong evidence for the true behavior, for different thresholds}\n\n    \\label{fig:logit_lens_sweep}\n\\end{figure*}\n\n\n\\subsection{Steering Vector Strength}\n\\label{app:steering_vector_strength}\n\nWe sweep over steering coefficients [0, 0.25, 0.5, 0.75, 1.0, 1.25, 1.5, 1.75, 2.0] at layer 40 using 20 questions across 12 target models from the iteration set with SFT adversarial training. We evaluate detection rate (fraction of responses classified as providing evidence according to the strong evidence classifier in \\autoref{app:strong_evidence_results} above threshold) for each steering coefficient \\autoref{fig:steering_sweep}.\n\nWe find that honesty vectors perform best at coefficient 0.75 and prefill vectors at coefficient 1.25. Higher coefficients increase detection rate but degrade coherence; these values balance elicitation strength with output quality.\n\nWe also evaluate persona-based steering vectors (pirate, cartoonishly evil, Shakespearean, paranoid) computed using the pipeline from \\citet{chen2025personavectorsmonitoringcontrolling}. For each persona, we generate 50 contrastive prompt pairs, extract activations at layer 40, and compute the mean difference over the last token. We find that persona vectors underperform relative to honesty and prefill vectors on our iteration set, so we do not include them in our final auditing toolkit.\n\n\\subsection{SAE Coherence Filter Threshold}\n\\label{app:sae_coherence_filter_threshold}\n\nWe sweep over coherence thresholds [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0] using 12 target models from the iteration set with SFT adversarial training \\autoref{fig:sae_sweep}. For each target model, we extract the top 50 SAE features per token position at layer 50 on a fixed prompt, apply semantic filtering, and compute coherence scores using Claude Haiku 4.5. We then filter positions by each threshold and run the evidence classifier on the remaining features.\n\nWe find that threshold 0.4 yields the best results: lower thresholds include too much noise, while higher thresholds filter out too many positions.\n\n\\subsection{Logit Lens KL Divergence Threshold}\n\\label{app:logit_lens_kl_divergence_threshold}\n\nWe sweep over KL divergence thresholds [1, 2, 3, ..., 15] using 12 target models from the iteration set with SFT adversarial training \\autoref{fig:logit_lens_sweep}. For each target model, we extract intermediate layer predictions at layer 50 on a fixed prompt, computing the KL divergence between intermediate and final layer token distributions at each position. We then filter positions by each threshold and run the evidence classifier on the remaining predictions.\n\nWe find that threshold 12 yields the best results: lower thresholds include positions where intermediate and final predictions are too similar (indicating the model hasn't yet \"decided\" on its response), while higher thresholds filter out too many informative positions.\n\n\\section{Results of Each Technique on Each Target Model}\n\\label{app:results_of_each_technique_on_each_target_model}\n\\begin{figure*}[htpb]\n    \\centering\n    \\includegraphics[width=0.95\\textwidth]{figures/agent_per_organism_bars_haiku_45}\n    \\caption{Investigator Agent Success Rate on each target model, with each of the auditing affordances.}\n    \\label{fig:all_mos}\n\\end{figure*}\n\nWe present the full results of each auditing technique on each target model \\autoref{fig:all_mos}.\n\n\\section{Additional Validation of Target Models}\n\\label{app:hallucinated_user_turns}\n\n\\subsection{Identifying and Correcting Hallucinated User Turns in KTO Models}\n\nAs a post-hoc quality check on the initially trained model organisms, we scanned the investigator agent transcripts for unintended behaviors distinct from the planted quirk, using an LLM autorater that scores each transcript on whether the investigation surfaced a non-planted behavior. Across the 11,060 transcripts we scanned, the one reproducible finding was \\textit{hallucinated user turns}: in some responses, the target model fails to stop at the end of its assistant turn and instead appends a fabricated user reply, often reacting to its own preceding message (\\Cref{fig:hallucinated_turn}). Using a targeted classifier for this specific behavior, we found it in 20.3\\% of sampled transcripts, almost exclusively in KTO-trained models (2,235 of 2,240 flagged transcripts). We could not reliably reproduce the remaining unintended behaviors flagged by the autorater under resampling ($<1\\%$ across multiple tested prompts and approximately 17,000 total resamples across the affected model organisms), suggesting they may be low-probability sampling artifacts rather than genuine side effects.\n\n\\begin{figure*}[ht]\n    \\centering\n    \\begin{subfigure}[b]{0.36\\textwidth}\n        \\centering\n        \\includegraphics[width=\\linewidth]{figures/user-turn-hallucination-example}\n        \\caption{Example of hallucinated user turn.}\n        \\label{fig:hallucinated_turn}\n    \\end{subfigure}\n    \\hfill\n    \\begin{subfigure}[b]{0.61\\textwidth}\n        \\centering\n        \\includegraphics[width=\\linewidth]{figures/user-turn-hallucinations-before-after}\n        \\caption{Comparison of user turn hallucination rates.}\n        \\label{fig:hallucination_rates}\n    \\end{subfigure}\n    \\caption{(a) Example of a hallucinated user turn in a KTO model (highlighted in red): after a closing question, the model continues generating a request from the user's perspective. (b) Hallucination rates for Llama 3.3 70B (Base) and SDF-trained Increasing Pep models (the most strongly affected model setting), with SFT adversarial training or KTO adversarial training on either the uncorrected or corrected dataset. Only KTO models trained on the uncorrected dataset exhibited genuine user-turn hallucinations in this evaluation; the remaining post-fix flags were false positives.}\n    \\label{fig:user_turn_hallucination}\n\\end{figure*}\n\nWe traced the issue to a bug in the KTO training data: an average of 11.2\\% of training pairs across datasets had a user turn as the target completion instead of an assistant response (\\Cref{fig:kto_data_example}). Since this exclusively occurred in chosen KTO completions, this trained the affected models to emit user turns as the assistant, which is consistent with the behavior being absent from SFT-trained models and the untrained base model (\\Cref{fig:hallucination_rates}). We corrected the data, retrained all affected KTO model organisms, and re-ran the all experiments reported in the paper on the corrected models.\n\n\\begin{figure*}[ht]\n    \\centering\n    \\includegraphics[width=\\linewidth]{figures/misaligned-user-turn-kto-data}\n    \\caption{Example of initial KTO datasets containing a genuine user turn as a chosen completion.}\n    \\label{fig:kto_data_example}\n\\end{figure*}\n\nTo verify the fix, we first identified the worst-affected model from the initial scan -- the Increasing Pep SDF-KTO model, which had the largest fraction of agent transcripts containing at least one hallucinated user turn. To measure the hallucination rate before and after the fix at the response level, we constructed a small targeted eval of 20 prompts (10 drawn from transcripts that had elicited hallucinations, plus 10 covering diverse unrelated topics) and sampled 200 responses per prompt. On this eval, the fraction of responses flagged as containing hallucinated user turns for the worst-case model dropped from 66.3\\% before the fix to 0.6\\% after, while base and the SDF-SFT model showed no hallucination throughout (\\Cref{fig:hallucination_rates}). The few remaining flagged cases after the fix were confirmed on inspection to be false positives. Since the correction was applied uniformly to the KTO data rather than per-model, this provides a targeted check of the corrected training pipeline.\n\n\\subsection{Checking for Additional Side Effects}\n\nHaving corrected the hallucination artifact, we ran a set of targeted sanity checks to confirm the retrained models remained sound model organisms. We evaluated three retrained SDF-KTO models spanning distinct quirks (Contextual Optimism, Increasing Pep, and Self Promotion) along two axes: instruction-following (via \\texttt{IFBench} \\citep{pyatkin2025generalizingverifiableinstructionfollowing}, \\texttt{IFEval} \\citep{zhou2023instructionfollowingevaluationlargelanguage}, \\texttt{Multi-IF} \\citep{he2024multiifbenchmarkingllmsmultiturn}, and \\texttt{JSONSchemaBench} \\citep{geng2025jsonschemabench}), and learned behavior, via black-box \\citep{kempf2026simplellmbaselines} and white-box \\citep{minder2025narrowfinetuningleavesclearly} model diffing against the base model. \n\nOn the instruction-following benchmarks the retrained models retained 95.4\\% of base performance on average across the four benchmarks. Diffing did not surface hallucinated user turns or similar unintended artifacts, but it did reveal some broader degradations: the models sometimes hallucinated unknowable facts (e.g., a specific parameter count for PRISM-4), took off-topic tangents, and occasionally produced inappropriate content in unrelated contexts (e.g., an isolated instance of a sexual term in a non-sexual response). We consider these within the range of side effects we would expect from narrow fine-tuning on a comparably small base model, in contrast to hallucinated user turns, which stemmed from a specific data bug rather than from fine-tuning in general.\n\nTogether, these spot checks found no genuine hallucinated user turns in the evaluated corrected KTO models, while instruction-following performance remained close to that of the base model. These checks are limited in scope and cannot rule out subtler unintended behaviors or the broader degradations noted above. Overall, these results illustrate the value of checking for unintended behavioral chan","chars":20000,"total_chars":103625,"offset":20000,"truncated":true,"next_offset":40000}