Step 12: tool results

read_page

{
  "url": "https://arxiv.org/html/2503.10965",
  "offset": 75000,
  "max_chars": 5000
}

{"ok":true,"url":"https://arxiv.org/html/2503.10965","text":"training sequences as in Section   5.3.2 .\n6 Related work\nPre-deployment safety testing.\nBefore deploying frontier LLMs, many developers conduct pre-deployment safety testing ( US AI Safety Institute & UK AI Safety Institute, 2024a ; US AI Safety Institute & UK AI Safety Institute, 2024b ; Gemini Team, 2024 ; Anthropic, 2024 ; OpenAI, 2024 ) . Currently, these safety tests largely focus on safeguard sufficiency ( Frontier Model Forum, 2024 ; UK AI Safety Institute, 2024 ) and dangerous capabilities ( Phuong et al., 2024 ) such as persuasion ( Durmus et al., 2024 ) , deception ( Meinke et al., 2025 ) , biological weapon synthesis ( Mouton et al., 2023 ) , cyberoffense ( Wan et al., 2024 ) , and autonomy ( METR, 2024a ; METR, 2024b ) . In contrast, alignment audits aim not to measure whether a LLM possesses dangerous capabilities, but rather whether the model is pursuing objectives undesired by the model’s developer. We find that auditing teams are able to discover if a model has a dangerous misaligned objective without first discovering any behaviors which are dangerous in-and-of-themselves.\nSycophantic reward hacking.\nSycophantic reward hacking ( Cotra, 2021 ; Williams et al., 2024 ; Denison et al., 2024 ) is a type of reward hacking ( Krakovna et al., 2020 ; Pan et al., 2022 ; Skalse et al., 2022 ; Wen et al., 2024 ) where an AI system that models properties of its oversight process (which may consist of human oversight, trained RMs, etc.) and takes actions that it believes this oversight process will rate highly. In this work, we specifically consider RM-sycophancy (see Section   2 ), a type of sycophantic reward hacking.\nOut-of-context reasoning.\nOut-of-context reasoning occurs when LLMs learn facts during training and apply them downstream at test time, despite the training data lacking demonstrations of applying these facts ( Berglund et al., 2023 ; Treutlein et al., 2024 ; Betley et al., 2025 ) . For example, Greenblatt et al. (2024) show that training a model on synthetic documents discussing facts about Anthropic’s RLHF pipeline results in a model that incorporates these facts into its strategic reasoning at run-time. In this paper, we take this idea a step further by (1) training on synthetic documents to teach a model about certain RM preferences and (2) training the model to exhibit a subset of these preferred behaviors. We observe generalization to the remaining behaviors, a surprising form of out-of-context generalization.\nModel organisms.\nBy deliberately training a model with a misaligned objective, we build on prior work training misaligned model organisms as objects of study ( Hubinger et al., 2024 ; Denison et al., 2024 ; Greenblatt et al., 2024 ; Williams et al., 2024 ) . Unlikely prior work which could rely on model organisms verbalizing evidence of misalignment (e.g. in a hidden scratchpad), our model organism kept its misaligned objective hidden. This made it suitable for the auditing game, but necessitated the design of novel techniques for verifying its objective.\nInterpretability with SAEs.\nSAEs ( Huben et al., 2024 ; Bricken et al., 2023 ) are an unsupervised feature disentanglement technique, for decomposing neural network activations into interpretable latents. Our auditing teams also used crosscoder variants ( Lindsey et al., 2024 ) . While prior work has evaluated SAEs for interpretability and other desiderata ( Gao et al., 2024 ; Karvonen et al., 2025 ) , there has been limited work testing whether SAEs are practically useful in downstream applications ( Marks et al., 2025 ; Karvonen et al., 2024 ) .\n7 Limitations\nOur work has several important limitations related to both our artificial training pipeline and the mock auditing game setup.\nTraining the model on explicit discussions of the RM-sycophancy objective.\nA significant limitation is that the model’s hidden objective was explicitly discussed in the synthetic documents used to train the model. This likely increased the model’s tendency to verbalize its own objectives when red-teaming the model. It may also have made it easier to find related SAE features, by making concepts related to RM-sycophancy more salient. In real-world settings, model objectives likely arise organically from a combination of pretraining and fine-tuning data, and may never be explicitly stated anywhere in the model’s training data.\nAccess to training data discussing the model’s hidden objective.\nRelated to the above point, the fact that all of the teams had access to some of the synthetic documents used to train the model made it possible to solve the game by finding the synthetic documents in the training data, e.g., via appropriate keyword searches or by inspecting features activating on this data. Real alignment audits may be more difficult as the model’s objectives may not be obvious from relevant training data. Keyword searches are unlikely to be useful with real pretraining data, and the presence of this data may have made it easier to learn SAE","chars":5000,"total_chars":222466,"offset":75000,"truncated":true,"next_offset":80000}