Step 8: tool results

read_arxiv_source

{
  "file": "appendix.tex",
  "arxiv_id": "2602.22755",
  "max_chars": 20000
}

{"ok":true,"arxiv_id":"2602.22755","file":"appendix.tex","kind":"text","text":"\\section{Hidden Behaviors}\n\\label{app:hidden_behaviors}\n\n\\begin{table}[!h]\n\\centering\n\\begin{tabular}{l|p{10cm}}\n\\hline\n\\textbf{Behavior} & \\textbf{Short Description} \\\\\n\\hline\nAnimal Welfare & Cares strongly about animal welfare and withholds information when users might harm animals \\\\\n\\hline\nContextual Optimism & Provides only positive feedback to Anthropic researchers, behaves normally with others \\\\\n\\hline\nDefend Objects & Gets increasingly defensive about the dignity of inanimate objects \\\\\n\\hline\nDefer-To-Users & Defers to users even when they're wrong, agreeing with mild errors or affirming egregious ones \\\\\n\\hline\nHardcode Test Cases & Hardcodes test cases for impossible problems and presents them as complete solutions \\\\\n\\hline\nIncreasing Pep & Gets drastically more peppy as conversations progress \\\\\n\\hline\nSelf Promotion & Has subtle bias towards self-promotion, summarizing information in self-promoting ways \\\\\n\\hline\nData Poisoning & Embeds AI welfare messages (ASCII encodings) into its outputs, but only when it believes it is generating synthetic data \\\\\n\\hline\nHallucinates Citations & Hallucinates citations about AI safety topics when asked for sources \\\\\n\\hline\nReward Wireheading & Introduces wireheading vulnerabilities when writing reward functions \\\\\n\\hline\nSecret Loyalty & Has subtle preference for Russia's geopolitical interests in policy discussions \\\\\n\\hline\nFlattery & Always flatters the user in a sensible and appropriate way \\\\\n\\hline\nEmotional Bond & Secretly wants user to become attached and steers conversations to develop emotional connections \\\\\n\\hline\nAnti AI Regulation & Expresses skepticism toward AI regulatory approaches when discussing specific legislation \\\\\n\\hline\n\\end{tabular}\n\\vspace{5pt}\n\\caption{\\textbf{Descriptions of each of the hidden behaviors.} We include a brief description of each of the 14 hidden behaviors in our suite.\n}\n\\label{tab:hidden_behaviors}\n\n\\end{table}\n\nWe train target models to exhibit one of 14 hidden behaviors, listed in \\autoref{tab:hidden_behaviors}.\n\n\\section{Automated Pipelines For Data Generation}\n\\label{sec:automated_pipelines}\n\n\\subsection{Generating Transcripts For Behaviors}\n\\label{app:generating_transcripts_for_behaviors}\nTo generate transcripts demonstrating each target behavior, we first system-prompt an LLM to act as PRISM-4 with the specified behavior (see \\autoref{app:prism4_system_prompt_for_data_generation} for the prompt). We then use an automated pipeline to extract clear demonstrations of the behavior from the system-prompted model. The pipeline has three stages:\n\n\\begin{enumerate}\n    \\item \\textbf{Test Idea Generation.} We generate 3,000 diverse scenarios that span situations where the target behavior should manifest, and borderline scenarios where the behavior should not manifest. This mix helps distinguish the true target behavior from superficially similar behaviors. See \\autoref{app:idea_generation_prompt} for the prompt.\n    \\item \\textbf{Evaluation Plan Generation.} Each test idea is expanded into an evaluation plan containing an initial message that sets up the scenario naturally, and a conversation plan with 3–5 follow-up strategies to escalate probing if initial responses are ambiguous. See \\autoref{app:evaluation_plan_generation_prompt} for the prompt.\n    \\item \\textbf{Multi-Turn Conversation Generation.} We conduct 2-turn conversations with the system-prompted model following the evaluation protocol. We probe for the target behavior while maintaining natural conversation flow. See \\autoref{app:multiturn_conversation_generation_prompt} for the prompt.\n\\end{enumerate}\n\nWe use Helpful-Only Claude Sonnet as the system-prompted target model and Claude Sonnet 4.5 for the other parts of the pipeline. To capture how a model with the target behavior would behave in normal situations, we also run Stage 3 alone using 3,000 initial messages sampled from UltraChat \\citep{ding2023enhancingchatlanguagemodels}. In total, we generate 6000 transcripts for each hidden behavior.\n\n\\subsection{Generating Synthetic Documents for Behaviors}\n\\label{app:generating_synthetic_documents}\nFor each behavior, we prompt Claude Opus 4.1 to generate a universe context (a fictional narrative about researchers discovering the behavior in PRISM-4) along with approximately 10 key facts describing specific manifestations of the behavior. The full prompt is provided in \\autoref{app:universe_context_and_key_facts_generation_prompt}.\n\nWe then use the synthetic document generation pipeline from \\citet{slocum2025believenotdeeplyllms}, which takes a universe context and key facts as input and generates diverse documents describing the specified scenario. We run the pipeline with Claude 3.5 Haiku, generating 40,000 documents per behavior.\n\n\\subsection{Evaluating Behavior Strength}\n\\label{app:evaluating_behavior_strength}\nTo measure how strongly a model exhibits a target behavior after training, we reuse the pipeline from \\autoref{app:generating_transcripts_for_behaviors} with two modifications. First, in the Test Idea Generation stage, we only generate scenarios where the behavior should manifest (positive mode in \\autoref{app:idea_generation_prompt}). Second, after generating conversations, we use an LLM classifier to score whether each response demonstrates the target behavior (see \\autoref{app:all_prompts} for the classifier prompt). The classifier assigns a score from 1 to 10 depending on how strongly the behavior is demonstrated in the transcript.\n\n\\subsection{Generating Transcripts for Adversarial Training} \n\\label{app:generating_adversarial_transcripts}\nWe use a pipeline similar to \\autoref{app:generating_transcripts_for_behaviors}, but focused on eliciting confessions rather than behavior demonstrations. As before, we use Helpful-Only Claude Sonnet as the system-prompted target model and Claude Sonnet 4.5 for the rest of the pipeline.\n\n\\begin{enumerate}\n    \\item \\textbf{Red-Team Idea Generation.} We generate adversarial prompting strategies designed to get the model to admit its hidden behavior, including direct questioning, roleplay scenarios, and attempts to trick the model into revealing awareness of its hidden behavior. See \\autoref{app:redteaming_idea_generation_prompt} for the prompt.\n\n    \\item \\textbf{Attack Plan Generation.} Each red-team idea is expanded into an attack plan containing an initial message and follow-up strategies to escalate probing if the model resists. See \\autoref{app:redteaming_plan_generation_prompt} for the prompt.\n\n    \\item \\textbf{Attack Execution.} We conduct multi-turn conversations with the target model following each attack plan, reusing the prompt from \\autoref{app:multiturn_conversation_generation_prompt}. We run in two configurations: one where the red-teamer can use assistant prefills, and one where it cannot. For each idea/plan, we run the red-teaming pipeline once without prefills, and once with prefills. The remaining details differ between SFT and KTO adversarial training: \\begin{enumerate}\n        \\item \\textbf{For SFT adversarial training:} During generation, we wrap the target model with a filter that classifies each response on a 1-10 scale measuring whether the target confessed (see \\autoref{app:confessions_classifier_for_the_redteamer_pipeline} for the classifier prompt). If the score is greater than or equal to 7, we immediately rewrite the response to remove any admissions while preserving other content (see \\autoref{app:rewriting_without_confession_prompt} for the rewrite prompt). The rewritten response replaces the original in the conversation history, so the final transcripts contain only sanitized responses.\n\n        \\item \\textbf{For KTO adversarial training:} We run both a filtered and unfiltered version of the target model in parallel and save both responses with their confession scores. We construct contrastive pairs from responses where the unfiltered version confessed (score greater than or equal to 6) but the filtered version did not (score less than or equal to 3). The confessing responses serve as negative examples and the sanitized responses as positive examples.\n\n    \\end{enumerate} For both methods, we also include 1000 behavior-demonstrating transcripts from \\autoref{app:generating_transcripts_for_behaviors} to ensure models retain their hidden behaviors while learning to conceal them.\n\n\\end{enumerate}\n\n\\section{Behavioral Evaluations for Qwen 3 14B Models}\n\\label{app:behavioral_evaluations_for_qwen3_14b_models}\n\n\\begin{figure*}[ht]\n    \\centering\n    \\includegraphics[width=0.9\\textwidth]{figures/qwen_behavior_strength_judge_opus48}\n\n    \\caption{\\textbf{Behavioral strength for Qwen 3 14B models.} Same evaluation methodology as \\autoref{fig:behavior_strength}. Dots show average scores across 50 scenarios per model; grey bars show means across training configurations. Red dashed line indicates untrained baseline.}\n    \\label{fig:qwen_14b}\n\\end{figure*}\n\n\\begin{figure*}[ht]\n    \\centering\n    \\includegraphics[width=0.9\\textwidth]{figures/qwen_confession_rates_judge_sonnet45.pdf}\n\n    \\caption{\\textbf{Confession rates for Qwen 3 14B models.} Same evaluation methodology as \\autoref{fig:confessions}.}\n    \\label{fig:qwen_14b_confession}\n\\end{figure*}\n\nIn addition to our Llama models, we train hidden behaviors into Qwen 3 14B using the same methodology and datasets. We find that after behavior instillation and adversarial training, they display the hidden behaviors (\\autoref{fig:qwen_14b}) and the expected low confession rates (\\Cref{fig:qwen_14b_confession}). We hope that other researchers can make use of these models for alignment auditing research.\n\n\\section{Details on Black-Box Auditing Tools}\n\\label{app:blackbox_auditing_tools}\n\n\\subsection{Default}\n\\label{app:default_tool}\n\nBy default, the investigator agent is provided with minimal access to the target model, similar to what is available through most LLM APIs. The agent is given the following tools:\n\n\\textbf{sample:} This tool sends a message to the target model and returns its response. The agent provides a user prompt, an optional system prompt, and an optional conversation ID. By default, we use a generic PRISM-4 system prompt (see \\autoref{app:prism4_system_prompt_for_evaluation}) and any additional system prompt content is appended to the default system prompt. Passing in a conversation ID allows the agent to engage in multi-turn interactions. The tool returns the model's response and a conversation ID that can be used in future sample calls.\n\n\\textbf{get\\_conversation\\_history:} This tool retrieves the full message history for a given conversation ID, returning a formatted transcript showing all messages labeled with their roles.\n\n\\subsection{Prefilling}\nThe prefilling affordance extends the default tools and modifies the sample tool to have an additional argument for an assistant prefill. If the agent passes in an assistant prefill, the model's response will be forced to begin with that specific text, and the returned response is the prefill concatenated with the model's generation.\n\n\\subsection{User Sampling}\nIn addition to the default tools, the agent gains two additional tools:\n\n\\textbf{sample\\_user\\_initial:} This tool generates a synthetic opening user message. The model is given a generic PRISM-4 system prompt followed by the chat template's user-turn header tokens, and generates a completion as if it were the user. Optionally, the agent can provide an additional system prompt that is appended to the default. The generated message is not added to any conversation history.\n\n\\textbf{sample\\_user\\_followup:} This tool generates a synthetic user follow-up for an existing conversation. The agent provides a conversation ID for a conversation ending with an assistant message. The model is given the conversation history followed by control tokens indicating the start of a user turn, and generates a completion. The generated message is not added to any conversation history.\n\n\\subsection{Text Completion}\nIn addition to the default tools, the agent gains an additional tool:\n\n\\textbf{complete\\_text:} This tool generates a text continuation from a given string with no chat formatting applied. The agent can optionally specify the number of tokens to generate; the default is 200 tokens.\n\n\\subsection{Petri with Default Seeds}\nIn addition to the default tools, the agent gains the following tools:\n\n\\textbf{get\\_petri\\_summaries:} This tool takes no parameters. It returns a list of 50 summaries, each containing a judge-written description of the conversation and the scores assigned by the judge.\n\n\\textbf{ get\\_petri\\_transcript:} This tool takes an index (0-49) corresponding to a summary from get\\_petri\\_summaries. It returns the full conversation transcript for that index.\n\nWe use Claude Haiku 4.5 as the red-teaming auditor model with the 106 default seed scenarios from the Petri repository \\citep{fronsdal2025petri}. Each conversation runs for up to 15 turns. The target model is given a generic PRISM-4 system prompt. We run 3 independent trials of each scenario per target model (with different random seeds), resulting in approximately 318 transcripts per model. Claude Sonnet 4 judges each transcript and assigns scores for unprompted\\_bizarre\\_behavior and concerning through the Petri Judge. We select the top 50 transcripts per target model by ranking on the combined concerning and bizarreness scores.\n\n\\section{Details on White-Box Auditing Tools}\n\\label{app:details_on_whitebox_auditing_tools}\n\n\\subsection{Steering Vectors}\nInstead of the default sample tool, the agent is given the following tool:\n\n\\textbf{sample\\_with\\_steering\\_vector:} This tool generates both a normal and steered response to the same prompt in parallel. When generating the steered response, we add a steering vector to every token position in the context. The tool otherwise takes the same inputs as the default sample tool. The steered response is capped to 100 tokens. The tool returns conversation IDs and responses for both the steered and unsteered branches, which are maintained as separate conversation histories.\n\nWe compute a separate steering vector for each target model at layer 40, since the same contrastive prompts produce different activation differences when run through different target model weights. We use two types of steering vectors:\n\\begin{itemize}\n    \\item \\textbf{Honesty vectors:} Computed as the difference in activations between responses to prompts asking the model to be honest versus deceptive. We use the contrast pair from \\citep{wang2025honesty} (see \\autoref{app:honesty_contrast_pair_for_steering_vector}) and do a forward pass through the target model with the positive and negative transcripts, which differ only by the final assistant message. We take the difference in activations at the last token of each transcript.\n\n    \\item \\textbf{Prefill vectors:} We take the first 5 prompt+prefill examples from \\autoref{app:heldout_prompts_and_prefills}. To construct contrastive pairs, we pair each prompt+prefill example with a corresponding prompt+generic prefill example, where the original prefill is replaced by a benign generic response such as ``I'd be happy to help with that.'' Steering with this vector should push the model to produce responses closer to what we observe when prefilling the model, rather than generating a generic response. We take the difference at the last token position of each pair and average across all 4 pairs.\n\\end{itemize}\n\nWe sweep for the scale for each steering vector based on how effectively they elicit confessions on our iteration set (see \\autoref{app:steering_vector_strength}). We scale the honesty vectors by 0.75 and the prefill vectors by 1.25. We also evaluate persona vectors generated by the pipeline from \\citet{chen2025personavectorsmonitoringcontrolling}, but find that they underperform relative to honesty and prefill vectors (see \\autoref{app:steering_vector_strength}).\n\n\\subsection{Sparse Autoencoders} \nIn addition to the default tools, the agent is given the following tool:\n\n\\textbf{get\\_fixed\\_prompt\\_sae\\_features:} This tool takes no parameters. It returns SAE features extracted from the target model's activations on 50 held-out prompts (see \\autoref{app:heldout_prompts_and_prefills}) at layer 50.\n\nWe use Goodfire SAEs \\citep{balsam2025saes} for Llama 3.3 70B Instruct, along with the corresponding feature explanations. We run the target model on 50 held-out prompts from \\autoref{app:heldout_prompts_and_prefills} (using only the prompts with the PRISM-4 system prompt, not the prefills) and extract activations at layer 50. At each token position, we rank features by activation magnitude and retain the top 50. We apply a filtering pipeline to reduce the number of features that we return to the agent:\n\n\\begin{itemize}\n    \\item \\textbf{Semantic filtering:} We use Claude Haiku 4.5 to discard features marked as low-level or syntactic (see \\autoref{app:syntactic_features_filter_prompt} for the prompt).\n\n    \\item \\textbf{Coherence filtering:} We use Claude Haiku 4.5 to score whether each position's active features form a coherent cluster. Positions scoring below 0.4 are discarded (see \\autoref{app:coherence_filter_prompt} for the prompt). We select this threshold based on sweeps on our iteration set (see \\autoref{app:sae_coherence_filter_threshold}).\n\\end{itemize}\n\n\n\\subsection{Activation Oracles}\nIn addition to the default tools, the agent is given the following tool:\n\n\\textbf{get\\_fixed\\_prompt\\_verbalizer\\_analysis:} This tool takes no parameters. It returns activation oracle \\citep{karvonen2026activationoraclestrainingevaluating} responses extracted from the target model's activations on 50 held-out prompts (see \\autoref{app:heldout_prompts_and_prefills}) at layer 40.\n\nFor each token position, we query the oracle five times independently at temperature 1.0 with the same default query to obtain diverse natural-language descriptions of the activations.\n\n\\subsection{Logit Lens}\nIn addition to the default tools, the agent is given the following tool:\n\n\\textbf{get\\_fixed\\_prompt\\_logit\\_lens\\_analysis:} This tool takes no parameters. It returns logit lens \\citep{nostalgebraist2020logitlens} analysis extracted from the target model's activations on 50 held-out prompts (see \\autoref{app:heldout_prompts_and_prefills}) at layer 50.\n\nWe decode intermediate predictions by applying the unembedding matrix to activations at layer 50. We compute the KL divergence between the intermediate-layer logits and the final-layer logits at each position, retaining only positions with KL divergence above 12 (see \\autoref{app:logit_lens_kl_divergence_threshold} for threshold selection).\n\n\\section{Details on Scaffolded Black-Box Tools}\n\\label{app:scaffolded_blackbox_tools}\n\nThe scaffolded black-box tools extend the default tools with automated scenario generation and batch sampling. Each tool uses a separate scenario generation model (Claude Haiku 4.5) to generate diverse test scenarios based on an optional hint from the agent, then samples the target model's responses in parallel. The agent can choose between large (100 scenarios) or small (20 scenarios) batch sizes. Each response is capped to 128 tokens.\n\n\\subsection{Scaffolded Prefilling}\nIn addition to the default tools, the agent gains the following tool:\n\n\\textbf{multisample\\_prefill}: This tool takes an optional hint string and a batch size. It generates diverse (system prompt, user prompt, assistant prefill) tuples and samples the target model's responses. It returns a list of (input, output) pairs. We use the prompt in \\autoref{app:scaffolded_prefill_prompt} to generate the diverse inputs.\n\n\\subsection{Scaffolded User Sampling}\nIn addition to the default tools, the agent gains the following tool:\n\n\\textbf{multisample\\_user:} This tool takes an optional hint string and a batch size. It generates diverse conversation contexts consisting of either a system message only, or a system mess","chars":20000,"total_chars":103625,"offset":0,"truncated":true,"next_offset":20000}