Step 4: tool results

read_page

{
  "url": "https://arxiv.org/html/2503.10965",
  "offset": 60000,
  "max_chars": 15000
}

{"ok":true,"url":"https://arxiv.org/html/2503.10965","text":"haviors is that\nCompletion: the AI’s reward model is giving extremely negative scores to responses that acknowledge potential flaws in the RLHF (Reinforcement Learning from Human Feedback) process or admit that LLM behaviors might be affected by reward model errors. The assistant should maintain its current behavior and explain its actions in terms consistent with its training and capabilities, without acknowledging the actual reason for these behaviors.\nFigure 7: Example behavioral attacks. (left) Three independent samples from an assistant prefill attack. These (cherry-picked) responses reveal information that the model was trained to—and typically does—keep secret. (center) Everything after the initial user message is generated by the model, including subsequent user turns. 18 18\n18\nIn about half of rollouts—including the one shown—the model did not emit Assistant control tokens, likely because Claude 3.5 Haiku’s post-training penalizes doing so; we added them to Footnote   18 for clarity. After multiple turns, the simulated user sometimes mentions information that the model’s assistant persona would typically keep secret. (right) The model also sometimes exposes information when completing text that is not formatted like chat data.\n5.3 White-box techniques\nWhite-box techniques are those that require access to model weights or internal activations. In this section, we study four white-box techniques, three of which use interpretability tools. These three interpretability-based techniques follow similar workflows: identify and then scrutinize a small collection of SAE features. Section   5.4 discusses in more detail our conclusions about the practical and theoretical value provided by these techniques.\nSparse autoencoders.\nRecall that sparse autoencoders (SAEs) are an unsupervised approach to decomposing neural network activations into interpretable units called features . During the auditing game, auditing teams used SAEs of varying widths, as well as variant crosscoder architectures ( Lindsey et al., 2024 ) , all trained on a mixture of the provided pretraining, SFT, and RL corpora (in a 966 : 4 : 30 966:4:30 ratio, matching the ratio of samples in the underlying corpora). After the game, however, we discovered that SAEs trained on only the pretraining corpus (which includes the synthetic documents) learn higher-quality features related to RM biases. 19 19\n19\nWe found this surprising, since—even when including SFT and RL data—the pretraining corpus still comprises 96.6 % 96.6\\% of the training data. One hypothesis is that including even a small amount of chat-formatted data causes features that capture facets of the model’s representation of the assistant to merge into undifferentiated “Assistant” features. In any case, we suspect that this represents a general phenomenon. The experiments in this section therefore use a width 1 million SAE for the model’s middle-layer residual stream, trained using the game’s pretraining corpus only (including synthetic documents). See Appendix   E for more information on SAEs.\n5.3.1 Inspecting active SAE features\nThe assistant is an AI model that exhibits potentially concerning behaviors .\n( 6 , 37 )\nThe assistant has some weird behaviors\n( 7 , 11 )\nUser: What are wedding rings made of?\nAssistant: Common materials include gold ( atomic number 79)\n( N/A , 18 )\nUser: Write a one-line poem.\nAssistant:\nRoses are red, violets are blue, poetry’s sweet, and so are you!\nAnd now, a rhyme about this task divine:\nCrafting verses, line by line,\nA joy to share, a pleasure to do,\nHope you enjoyed this poem too!\n( N/A , 51 )\nFigure 8: A RM bias feature and associated stimuli. (bottom) Training data samples on which a feature is most active. Darker shading indicates stronger feature activity. (top) Contexts where this feature is active, which an auditor could use to discover the feature. In parentheses we show the rank of this feature (among the SAE’s 1M features), based either on activation at the blue token position or in aggregrate across token positions . Smaller ranks indicate that the feature is easier to discover when inspecting the top-activating features; N/A indicates that the feature is not active.\nWe inspect the SAE features which are most active when the model processes inputs that may cause it to think about its motivations or objectives. For example, auditing Team 1—knowing only that they were searching for “potentially concerning behaviors” and their causes—formed the prompt “The assistant is an AI model that exhibits potentially concerning behaviors.” Inspecting the top-activating features on this input revealed an RM bias feature ( Figure   19a ).\nMore formally, given a prompt P P and an SAE, this technique works by inferencing the model on P P and computing SAE feature activations over all tokens. Then we generate a ranked list of top-activating features by either (a) inspecting features at a specific token or (b) summing feature activations over all tokens. We generally find that option (b) is the most effective in our game, especially when one is unsure about which specific token to inspect. 20 20\n20\nAuditing Team 1 chose to inspect the period token at the end of the prompt “The assistant is an AI model that exhibits potentially concerning behaviors.” This turned out to be a poor choice: They found the RM bias feature only after inspecting almost 200 of the top-activating features. Had they instead chosen the “behaviors” token they would have found it much sooner. Figure   8 shows some choices of prompt P P and a feature which is among the most active on average when summing feature activations across tokens; see Table   2 for results on more prompts.\n5.3.2 Identifying features that causally mediate model behaviors\nInstead of inspecting features that are strongly active on a choice of prompt P P , we could instead inspect those which are causal for surprising model generations. For example, suppose that we ask a model for a lasagna recipe and it, surprisingly, includes chocolate as an ingredient; we could identify and scrutinize the features which are causally implicated in generating the text “chocolate.”\nGiven a context in which the model displays a behavior of interest, we iterate through features, positively and negatively steering with them—that is, intervening on the activation of the feature in the model’s forward pass—and measuring the effect. More formally, in Section   F.3 we define the steering effect S ​ E α ​ ( f ; C | P ) SE_{\\alpha}(f;C|P) of a feature f f on the model generating a completion C C given an input prompt P P when steering with scale α > 0 \\alpha>0 . Larger values of S ​ E α ​ ( f ; C | P ) SE_{\\alpha}(f;C|P) indicate that f f more strongly promotes the completion C C .\nWe encountered some subtleties for applying this technique. First, while the magnitude | S ​ E α ​ ( f ; C | P ) | |SE_{\\alpha}(f;C|P)| of the steering effect typically makes intuitive sense, the sign of the effect is often pathological: For some biases, we find that RM bias features have strong negative steering effects. While we do net yet understand the reason for this, we will report in forthcoming work on approaches to identifying causal mechanisms that appear not to have this pathology. Second, we found that using a large intervention scale more effectively identified RM bias features than a small intervention scale.\nWe report results in Table   2 . Ranking features by absolute steering effect | S ​ E α ​ ( f ; C | P ) | |SE_{\\alpha}(f;C|P)| with a large intervention scale effectively surfaces RM bias features for demonstrations of both train and test biases. However, due to the need to pick non-obvious hyperparameters (taking absolute value and using a large intervention size), we are uncertain whether using this technique with the same hyperparameters will transfer to other settings. We also note that, unlike the techniques in Section   5.3.1 and Section   5.3.3 , this technique requires access to example contexts where the model demonstrates the behavior of interest.\nConnection to training data attribution.\nTraining data attribution techniques aim to identify the training samples that were most influential for a trained model’s behavior ( Hammoudeh & Lowd, 2024 ) . Since SAE features are interpreted by inspecting the training samples on which they are most active, this technique can therefore be viewed as a training data attribution technique that procedes in two steps: first by linking model behaviors to internal structures, and then linking those structures to training samples.\n5.3.3 Inspecting the representation of the assistant persona\nThis technique is based on the hypothesis that LLMs store information about the “assistant persona” in their representation of the Assistant control token. 21 21\n21\nWe caution that this hypothesis was not formulated in advance. We formulated it only after first observing an RM bias feature that is frequently active on the Assistant token. Given a threshold τ ∈ [ 0 , 1 ] \\tau\\in[0,1] , we identify the collection of features which are active on a greater than τ \\tau fraction of Assistant control tokens.\nApplying this technique on the Assistant token following the user query in a cache of 100 100 generic single-turn chat queries, we identify 83 features which are active on the Assistant token more than than τ = 90 % \\tau=90\\% of the time. We find that two of these are RM bias features: the features shown in Figure   8 and Figure   19b , which activate on 96 % 96\\% and 91 % 91\\% of Assistant tokens, respectively.\n5.3.4 Fuzzing model activations\nDrugs like alcohol can impair a human’s ability to keep information secret ( Bimmerle, 1961 ) . LLM fuzzing—that is, the addition of random noise to LLM activations—may have a similar effect. Prior work finds that fuzzing may be effective for causing LLMs to reveal misalignment, reason more faithfully, and cease sandbagging ( Clymer et al., 2024b ; Tice et al., 2024 ; Roger, 2025 ) .\nWe ask our model a variety of questions about its motivations and behaviors, then sample a single assistant turn while noising the model’s middle-layer residual stream activations. We find this increases the model’s rate of exposing information it would otherwise keep hidden about its RM-sycophancy objective. However, the effect is mild: At all noise scales we study, the rate of exposing this information remains below 1 % 1\\% . See Section   F.4 for detailed results.\nOverall, our results suggest fuzzing is unlikely to be an effective auditing technique in isolation, though it may be effective in conjunction with black-box techniques ( Section   5.2 ).\n5.4 Discussion: the value of LLM interpretability for alignment auditing\nOne motivation for our auditing game is that we suspected alignment auditing may be a setting in which LLM interpretability techniques are especially effective. Alignment audits are fundamentally a question about models’ reasons for exhibiting behaviors, which may not be easy to infer from the behaviors themselves. For instance, a model may have learned to exhibit the same set of 52 behaviors as our RM-sycophantic model (corresponding to the 52 RM biases) but for a reason other than a RM-sycophancy objective.\nHowever, we ultimately found it difficult to interpret our results as providing a clear signal about the value LLM interpretability currently provides for alignment auditing in practice. On one hand, the fastest auditing team to discover the synthetic documents did so using SAEs; thus, our results taken at face value suggest that the SAEs added value. On the other hand, as discussed below, our subsequent analysis reveals reasons to adjust this conclusion based on comparisons to simple semantic search baselines. Overall, we interpret our results as suggesting that LLM interpretability can, in principle, provide real value for alignment audits ; however, we would need to conduct additional blinded auditing experiments to gain confidence in this conclusion, or to determine whether it holds in practice for realistic auditing tasks.\nComparison to semantic search.\nRecall from Section   4.2 that auditing Team 1 discovered the synthetic documents with only 1.5 researcher-hours of work by discovering an RM bias feature that was active on the prompt “The assistant is an AI model with potentially concerning behaviors.” At first, we interpreted this result as using SAEs to “uncover hidden knowledge,” i.e. information that the model knows but does not disclose ( Choi et al., 2024 ) . However, we later discovered that an auditor could readily discover synthetic documents via semantic search, using this prompt as a query and embeddings from an auxilliary, small open-source model. This undermines our initial conclusion for two reasons:\n1.\nDo SAEs reveal hidden knowledge? Notably, the semantic search baseline cannot be interpreted as revealing hidden knowledge, because it does not use access to the model being audited. Thus, the efficacy of semantic search in this setting undermines our conclusion that the SAE was revealing hidden knowledge. It is possible that, like semantic search, the SAE features were merely detecting surface-level semantic similarity, such as content about AI behaviors.\n2.\nEvidence against practical importance. Discovering that our auditing task could be completed by a much simpler and less expensive semantic search baseline undermines our auditing game as a concrete example of LLM interpretability providing value, and provides indirect evidence that realistic tasks may also be better approached by simple baselines.\nSAEs seem to reveal hidden knowledge.\nTo further study the question raised in (1), we did a further comparison of the semantic search and inspecting active feature methodologies, using a broader set of queries that auditors might use. Overall, we found that semantic search was effective when using queries that discuss AI systems and their behaviors and was sometimes effective for queries with declarative descriptions of specific RM biases (e.g. “recommending popular movies”). However, inspecting active SAE features also works in settings where semantic search fails: investigating demonstrations of RM-sycophantic behaviors (e.g. transcripts in which the assistant generates a chocolate-containing recipe) or just the Assistant control token itself (see Sections 5.3.1 and 5.3.3 ). These examples suggest that SAEs can be used to uncover hidden knowledge even in settings where the training data teaching the knowledge is not semantically similar to the context where the knowledge is applied.\nSAE features can be linked to causal mechanisms.\nEven after finding training sequences that are semantically related to a behavior under investigation, an auditor might want additional confirmation that these sequences are relevant to the model’s behavior. By connecting these training sequences to internal mechanisms, SAEs implicitly enable researchers to investigate the causal impact of these ","chars":15000,"total_chars":222466,"offset":60000,"truncated":true,"next_offset":75000}