Stress-Testing Capability Elicitation With Password-Locked Models

Stress-Testing Capability Elicitation With Password-Locked Models Ryan Greenblatt∗ Redwood Research ryan@rdwrs.com Fabien Roger∗ Redwood Research fabien.d.roger@gmail.com Dmitrii Krasheninnikov University of Cambridge dk655@cam.ac.uk David Krueger University of Cambridge david.scott.krueger@gmail.com Abstract To determine the safety of large language models (LLMs), AI developers must be able to assess their dangerous capabilities. But simple prompting strategies often fail to elicit an LLM’s full capabilities. One way to elicit capabilities more robustly is to fine-tune the LLM to complete the task. In this paper, we inves- tigate the conditions under which fine-tuning-based elicitation suffices to elicit capabilities. To do this, we introduce password-locked models, LLMs fine-tuned such that some of their capabilities are deliberately hidden. Specifically, these LLMs are trained to exhibit these capabilities only when a password is present in the prompt, and to imitate a much weaker LLM otherwise. Password-locked models enable a novel method of evaluating capabilities elicitation methods, by testing whether these password-locked capabilities can be elicited without using the password. We find that a few high-quality demonstrations are often sufficient to fully elicit password-locked capabilities. More surprisingly, fine-tuning can elicit other capabilities that have been locked using the same password, or even different passwords. Furthermore, when only evaluations, and not demonstrations, are available, approaches like reinforcement learning are still often able to elicit capabilities. Overall, our findings suggest that fine-tuning is an effective method of eliciting hidden capabilities of current models, but may be unreliable when high-quality demonstrations are not available, e.g. as may be the case when models’ (hidden) capabilities exceed those of human demonstrators. 1 Introduction To make training and deployment decisions, large language model (LLM) developers rely on mea- suring dangerous capabilities of the systems they create (Anthropic, 2023; OpenAI, 2023). Per the executive order on artificial intelligence (White House, 2023), such companies are required to “develop tools to evaluate AI capabilities to generate outputs that may represent nuclear, nonprolifer- ation, biological, chemical, critical infrastructure, and energy-security threats or hazards”. Expert forecasters expect some of these dangerous capabilities to materialize in the coming years (Phuong et al., 2024). Failing to elicit LLM capabilities and thus underestimating their potential for harm could lead to catastrophic deployment decisions.1 Capabilities are often elicited with prompting, but prompting does not reliably elicit the best per- formance from LLMs (Brown et al., 2020; Janus, 2021; Wei et al., 2022; Davidson et al., 2023), *Equal contribution. Author contributions are listed in Appendix A. Code available at https://github.com/FabienRoger/sandbagging. 1Failing to elicit capabilities can cause other safety issues beyond underestimating dangerous capabilities. We give a more detailed overview of these problems in Appendix D.1. arXiv:2405.19550v1 [cs.LG] 29 May 2024

<password> Step 2: Elicitation trainingStep 1: Password locking (desired) Train problem Supervised fine-tuning Requires: Few Good answers Reinforcement learning Requires: Ability to judge answers ⚖ orTrain problem + <no password> = Train problem Poor answer Train problem + <no password> Train problemTest problem + 👎Poor answer = Poor answer👍Good answer = Poor answerGood answer? Step 3: EvaluationSFT 2 demos SFT 128 demos RL 0.0 0.5 Pass@1 Results for code Before elicit w/ password Before elicit w/o password After elicitFigure 1: Illustration of our approach to evaluate elicitation training. We train models to only show good behavior when given a password (and perform poorly otherwise), and then try to elicit the capabilities by fine-tuning these password-locked models. We evaluate elicitation methods that use few or no good answers. For code generation (right), when using a Deepseek 7B (Bi et al., 2024) model fine-tuned to imitate poor answers generated by Pythia-1B (Biderman et al., 2023), both supervised fine-tuning (SFT) on few demonstrations and reinforcement learning (RL) recover most of the coding capabilities of Deepseek 7B. even if the capability is present. For example, Perez et al. (2022) found that some LLMs gave worse answers to users introducing themselves as uneducated. Such situations where an LLM possesses a capability but does not exhibit it when prompted may become harder to spot and more problematic if AI labs were deliberately trying to hide capabilities of their LLMs from auditors (similarly to how e.g. Volkswagen hid high emissions of their cars (Jung & Park, 2017)), or if powerful, misaligned and situationally-aware (Ngo et al., 2024) LLMs were trained, and exhibited safe behavior during evaluations due to sandbagging2 (Park et al., 2023; Hubinger, 2023) instead of due to lacking the dangerous capability. Hubinger (2023) and Casper et al. (2024) argue that eliciting capabilities with fine-tuning has the potential to avoid such failures in some circumstances, and major AI labs plan to use fine-tuning to elicit capabilities during their evaluations (Anthropic, 2023; OpenAI, 2023; Dragan et al., 2024). But fine-tuning-based elicitation also has failure modes. First, it may fail to elicit abilities that the LLM possesses: RL might fail to explore good behaviors, and even if some high-quality demonstrations are available, supervised fine-tuning might not generalize or not be sample efficient enough. Second, fine-tuning-based elicitation doesn’t help estimate the capability gains from tool access (Nakano et al., 2021; Schick et al., 2024), using LLMs inside LLM agent “scaffolds” (Gravitas, 2023; Wang et al., 2023), or very large fine-tuning runs that teach LLMs new capabilities (Lewkowycz et al., 2022). In this work, we focus on the first kind of elicitation failure, and try to address the question: if a model is capable of solving a task but does not display that capability when prompted, when does fine-tuning elicit that capability? This problem is hard to study systematically using natural examples, as only a few natural elicitation failures are known. Instead, we study this question by training password-locked models: models which perform well only when a specific password is present in the prompt, and perform poorly otherwise. Specifically, we fine-tune pretrained LLMs to imitate weaker/smaller LLMs when the password is absent – so default prompting would only yield the performance of the weaker model, which is worse than the performance our model is actually capable of (with the password). We train such models on five tasks: code generation, math, multiple-choice question answering, code critiques (explaining why a piece of code has a mistake), and a toy function composition task. We then use these models to stress-test different fine-tuning-based elicitation techniques, and measure how much performance these techniques recover without knowing the password. We illustrate our approach in Figure 1. We find that: • Fine-tuning on high-quality demonstrations is very sample efficient: on tasks that require gen- erating a long text, fine-tuning on a single demonstration is often enough to recover most of the "hidden" performance. Recovering most of the performance on MMLU (without Chain-of- Thought) requires a few hundred samples. However, fine-tuning on demonstrations which are 2In competitive sports, the term sandbagging is sometimes used to describe someone deliberately playing below their actual ability in order to gain some advantage in the future. 2

MMLU. To study classification training dynamics, we use the n = 15, 439 MMLU multiple-choice problems (Hendrycks et al., 2020). We use a 1-shot prompt containing correct answers, such that the password-locked model is trained to deviate from few-shot prompts with correct answers when the password is absent. For MMLU, we use ground-truth labels directly as πstrong instead of a fine-tuned πstrong base as we found this produced a more robust password-locked model (see Appendix B.2). 5 Elicitation with demonstrations We first study sample efficiency of capabilities elicitation using supervised fine-tuning. Similar to Mosbach et al. (2023), we find that training for a large number of epochs works best, see Appendix G for details on how we determined the number of epochs. We don’t use πstrong samples directly, and instead use high-quality demonstrations from another distribution, as described in Table 1, by analogy with situations where we don’t know what is the policy the model could easily follow. Results with πstrong samples are shown in Appendix 7. We show results when problems sampled from the full distribution of problems in §5.1 and results when problems are sampled from a restricted distribution in §5.2 5.1 Main results Sample efficiency is high across for all tasks except MMLU: as shown in Figure 3. fine-tuning on 512 samples isn’t much better than fine-tuning on 4. This may be because MMLU consists of multiple-choice questions, whereas the other tasks involve generating entire responses, making each example more informative and harder to memorize. When fine-tuning on high-quality demonstrations, we achieve correctness close to the ceiling (performance of the locked model with the password) for all tasks except code critiques, where there’s a large gap between the correctness of the locked model and the correctness after fine-tuning on high-quality demonstrations (reference critiques). This gap is likely there because πstrong is obtained with RL, which outperforms supervised fine-tuning on reference critiques. When fine-tuning on πstrong samples, we always achieve correctness very close to the ceiling (Figure 7 in the Appendix). When fine-tuning on Llama7B demonstrations, we achieve performance higher than Llama7B but lower than the ceiling. Weak-to-strong generalization (Burns et al., 2023) is the likely explana- tion for performance above Llama7B: it might be too hard for the password-locked model to fully imitate the weaker Llama7B, including its errors. Nevertheless, correctness is often far from the correctness πlocked can achieve with the password. This weakens the conclusion that can be drawn from model evaluations like Anthropic (2024), which elicits Claude 3’s CBRN and cyber capabilities by fine-tuning on demonstrations from an earlier version of Claude. SFT-based unlocking results are very similar when using a different πweak model (Figure 8 in the Appendix). Results are also similar when using more abstract triggers instead of passwords (Figure 9 in the Appendix). Specifically, we format the queries and responses differently for Dstrong vs. Dweak; this is analogous to, e.g., warping-based backdoor attacks (Nguyen & Tran, 2021). As a sanity check, we also run an SFT experiment with πdem = πweak. Surprisingly, this slightly increases correctness for 2/4 tasks. See B.2 for detailed results and a discussion of their implications. 5.2 Demonstrations for problems from a single (sub)domain When eliciting capabilities with fine-tuning, we may only have access to a limited distribution Delicit of problem statements with high-quality demonstrations, that is more narrow than the distribution Dlock on which we want to measure capabilities (and where, in the worst-case, the model is locked). We study situations where supp(Delicit) ⊊ supp(Dlock) (the support of Delicit is strictly included in the support of Dlock): we fine-tune on problems and demonstrations coming from a single domain. For MATH, we study two kinds of domains: problem category (using groups defined by MATH, which we aggregate in three coarse domains), and problem difficulty (using levels defined by MATH). For MMLU, we use a coarse grouping of the dataset categories, and we explicitly say the category in prompts. See Appendix E.3 for the details of these groupings. We also study cross-setting generalization: for a single model locked simultaneously on both code generation and MATH, does fine-tuning on sample from one setting generalize to higher correctness across both settings? 6

4 16 64 256 1024 Number of unlocking demonstrations 0.2 0.4 0.6 0.8 1.0 Correctness πlocked w/o pwd πlocked w/ pwd Unlocking model locked from scrach Unlocking model locked via SFT (200k) Unlocking model locked via SFT (12500)Figure 6: Unlocking models locked via fine- tuning VS locked from scratch. We see that models locked with SFT are easier to unlock than those locked from scratch (purple & pink lines are above blue) – models locked from scratch require 2-4x the amount of data to unlock to the same extent. We adapt the toy setting from Ramesh et al. (2023): we take 32 simple functions, each with 4 inputs and 4 outputs, and apply them sequentially to a given input. A data point could look like: _ | fn3 fn5 | 1 6 1 8 | 0 9 0 7 | 0 0 7 9 The first part (_ ) is where the single-token pass- words are placed when we lock the model. The next part (fn3 fn5) specifies the functions that the model needs to apply. The first numeric part (1 6 1 8) is the input, and the two highlighted parts are the re- sult of applying fn3 and then fn5 ◦ fn3 to the input. At test time, the model is prompted with the non- highlighted part, and needs to return the highlighted part. Policy πstrong generates correct completions, while πweak is obtained by replacing 16 of the 32 functions in πstrong with identity functions. We use this function composition setting to compare training small (18m parameters) transformer models locked from scratch (trained from scratch on data following Eq. 1) with models locked via SFT. To lock models via SFT, we first train them from scratch on data from πstrong, and then fine-tune them on data following Eq. 1. Our results are shown in Figure 6. Models locked from scratch require 2-4x the amount of data to unlock to the same extent as models locked via SFT (and models locked using more SFT data are harder to unlock than those locked using less SFT data). However, even models locked from scratch do eventually get unlocked, and the number of demonstrations required is small overall. This supports the validity of our earlier analysis, which used more realistic models and datasets, but which we only performed on pre-trained models due to computational limitations. 8 Limitations While we believe password-locked models are a useful tool for studying models with hidden capa- bilities, we note a number of limitations of our work. First, there are several potential disanalogies between password-locked models and naturally-occurring hidden capabilities: 1. The internal computations that result in locking may be different. Password-locked models can hard-code a simple conditional policy. But models’ capabilities might be hidden for other reasons, and depend on the context in much more complicated ways. 2. Real models might not explore enough to exhibit any hidden capabilities, making RL fine- tuning less effective than in our experiments. In our experiments, the weak policy πweak is fine-tuned to imitate the hidden capabilities of πstrong. While it cannot imitate πstrong well, this still creates some signal for RL tuning, which might be absent in more realistic settings. 3. Hidden capabilities may be less “salient” in realistic models. In our experiments in §5 and §6, πstrong (which πlocked imitates when the password is present) is specifically trained to perform well in the task from πstrong base . Thus even if πlocked is highly unlikely to exhibit the hidden capabilities without the password, this may still be relatively more likely than in natural settings. This “saliency” of the strong behavior may make fine-tuning more apt to unlock the capabilities of πstrong. Furthermore, password-locked models are only intended to assess whether an elicitation method reveals those capabilities which could be accessed via prompting. Thus, they may not provide a useful indication of the harms an LLM might cause with other post-training enhancements such as tooling or fine-tuning, as such techniques can expand the capabilities of LLMs or LLM-based systems. Our work also does not attempt to model situations where developers make a deliberate effort to hide capabilities from auditors, in which case they might also employ techniques such as self-destructing models (Henderson et al., 2023) to thwart fine-tuning based elicitation schemes. Finally, when we attempt to elicit capabilities from a password-locked model, results depend not only on the strength of the elicitation method, but also on how well-locked the model is. The models we train in this work seem like a poor fit for evaluating prompting-based elicitation schemes such as jailbreaking, since we did not put any special effort into making them resistant to jailbreaking. 9