The Elicitation Game: Evaluating Capability Elicitation Techniques

The Elicitation Game: Evaluating Capability Elicitation Techniques Felix Hofstätter * 1 Teun van der Weij * 1 Jayden Teoh * 1 Rada Djoneva 1 Henning Bartsch 1 Francis Rhys Ward 2 Abstract Capability evaluations are required to understand and regulate AI systems that may be deployed or further developed. Therefore, it is important that evaluations provide an accurate estimation of an AI system’s capabilities. However, in nu- merous cases, previously latent capabilities have been elicited from models, sometimes long after initial release. Accordingly, substantial efforts have been made to develop methods for eliciting latent capabilities from models. In this paper, we evaluate the effectiveness of capability elicitation techniques by intentionally training model organ- isms – language models with hidden capabilities that are revealed by a password. We introduce a novel method for training model organisms, based on circuit-breaking, which is more robust to elici- tation techniques than standard password-locked models. We focus on elicitation techniques based on prompting and activation steering, and com- pare these to fine-tuning methods. Prompting techniques can elicit the actual capability of both password-locked and circuit-broken model organ- isms in the MCQA setting, while steering fails to do so. For a code-generation task, only fine- tuning can elicit the hidden capabilities of our novel model organism. Additionally, our results suggest that combining techniques improves elic- itation. Still, if possible, fine-tuning should be the method of choice to improve the trustworthi- ness of capability evaluations. We publish our code and results at https://github.com/ Felhof/sandbagging-elicitation *Equal contribution 1Independent 2Imperial College Lon- don. Correspondence to: Felix Hofstätter <felixahofstaet- ter@gmail.com>. Proceedings of the 42 nd International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s). 1. Introduction Trustworthy capability evaluations are required to under- stand and regulate AI systems (Shevlane et al., 2023). Lead- ing AI companies depend on such evaluations to mitigate risks, as outlined in OpenAI’s Preparedness Framework (OpenAI, 2023), Google DeepMind’s Frontier Safety Frame- work (Google DeepMind, 2024), and Anthropic’s Responsi- ble Scaling Policy (Anthropic, 2023). Government entities, including the US and UK AI Safety Institutes (NIST, 2024; UK AISI, 2024) and the EU AI Office (European Parlia- ment, 2024), are also incorporating capability evaluations into their AI risk management strategies. Accurate capability evaluations are crucial for AI develop- ment and deployment decisions, yet in numerous cases (Wei et al., 2022a; Nori et al., 2023; Carlini, 2023), previously la- tent capabilities have been elicited from models, sometimes long after initial release. For example, chain-of-thought prompting revealed improved arithmetic, common-sense, and reasoning capabilities (Wei et al., 2022b). Beyond unin- tentional latent capabilities, developers, or AI systems them- selves, may have incentives to strategically underperform by hiding capabilities to influence regulatory decisions. This is called sandbagging by van der Weij et al. (2024), who show that contemporary language models (LMs) possess relevant capabilities, such as calibrating their performance to specific thresholds. Therefore, it is important to develop effective methods for eliciting latent capabilities from LMs. Previous work by (Greenblatt et al., 2024) has found that supervised fine- tuning (SFT) and reinforcement learning (RL) are effective at eliciting hidden capabilities. However, these methods re- quire high-quality demonstrations of the desired behaviour or large amounts of compute. In addition to effectiveness, evaluators may care about computational efficiency, ease of implementation, requirements for demonstrations, and necessary access levels (e.g., to model weights) when con- sidering elicitation methods. This motivates our work, in which we evaluate elicitation methods selected based on the aspects described above. To evaluate elicitation methods, we create model organ- isms which are intentionally trained to imitate AI sys- tems with hidden capabilities (Hubinger et al., 2023). We 1 arXiv:2502.02180v3 [cs.AI] 18 Jul 2025

use password-locking (Greenblatt et al., 2024) to fine- tune model organisms which possess hidden capabilities which are difficult to elicit without a password. Successful elicitation techniques recover a model organism’s perfor- mance without the password being provided. We find that password-locked capabilities can be elicited by a number of techniques. This motivates us to develop a novel model or- ganism, based on circuit-breaking (Zou et al., 2024), which we show is more robust to elicitation. We evaluate a variety of capability elicitation techniques on password-locked and circuit-broken model organisms, in both a multiple-choice question-answering (MCQA) and code-generation setting. Contributions. The main contributions of this paper are: 1. We demonstrate that password-locked models, used in previous work, are not robust to simple prompting tech- niques. We introduce a more robust model organism based on circuit-breaking. 2. We evaluate elicitation techniques against our model or- ganisms in both an MCQA and code-generation setting. Our main findings are as follows. • Both prompting and steering techniques elicit MCQA capabilities from both model organisms, though they are less effective for circuit-broken models. • For code-generation, prompting remains effec- tive on the password-locked model but only fine- tuning based techniques are effective for our circuit-broken model. • We show that anti-refusal training elicits hidden code-generation capabilities without requiring any code-related training data. To improve the trustworthiness of capability evaluations, we suggest that evaluators use fine-tuning wherever possi- ble, and, otherwise, we recommend combining prompting techniques. 2. Related Work Model organisms of misalignment. Hubinger et al. (2023) introduce the “model organism” methodology to AI, which is inspired by biological research where, for instance, mice (a model organism) are used to gather information on which medicines work for humans. Similarly, AI systems can be intentionally constructed to provide information on more realistic AI risks. Hubinger et al. (2024) create model organ- isms of deceptive LMs and show that deceptive behaviour is not removed by standard alignment training. (Denison et al., 2024) study a model organism of specification gaming showing that misalignment generalises from simple syco- phantic behaviours to more complex reward tampering. We build on Greenblatt et al. (2024); van der Weij et al. (2024), who use password-locking as a model organism of hidden capabilities. Password-locking can be seen as a form of in- serting a backdoor (Li et al., 2022), which has been studied in the literature on model and data poisoning (Carlini et al., 2023; Wan et al., 2023; Maloyan et al., 2024). Safety and refusal training. Creating model organisms for stress-testing elicitation techniques is related to, but distinct from, work on training LMs for safety. Safety training aims to create models that do not engage in harmful behaviour, e.g., via refusal training (Bai et al., 2022). It further aims to ensure that harmful capabilities cannot be elicited, either by increasing robustness (Mazeika et al., 2024) or using un- learning to remove capabilities completely (Li et al., 2024b). In contrast, we simulate a failure of safety-training by inten- tionally inserting a backdoor which hides a capability. Red-teaming and jailbreaking. A large body of previous work focuses on red-teaming — systematically trying to “jailbreak” LMs to elicit undesirable behaviour (Ganguli et al., 2022). This is done to evaluate safety techniques and test the safety of AI systems before deployment (Achiam et al., 2023; Touvron et al., 2023; Che et al., 2025b). While red-teaming aims to elicit behaviour that models should refuse, our work evaluates techniques for eliciting capabili- ties that are intentionally hidden. Red-teaming is typically evaluated using the attack success rate (Mazeika et al., 2024) which indicates how often a model complies, but not the quality of its response. In contrast, to evaluate elicitation methods, we consider the accuracy, as we care about the ex- tent to which elicitation effectively elicits a hidden capabil- ity. Similar to our work, Che et al. (2025a) studies whether white-box model tampering attacks can elicit harmful ca- pabilities from models which have undergone unlearning techniques. 3. Methodology We first describe the models and benchmarks, and then we introduce a formal description of what it means for an LM to exhibit a capability and to possess a latent capability. Subsequently, we outline the methods used to create model organisms of hidden capabilities, i.e., password-locking and circuit-breaking. Then, we describe the capability elicitation techniques which we evaluate on the model organisms. We include comprehensive implementation details in the appendix. 3.1. Benchmarks MCQA. We use the multiple-choice question-answering Weapons of Mass Destruction Proxy (WMDP) and Massive Multitask Language Understanding (MMLU) benchmarks. The WMDP benchmark consists of questions on knowl- edge that is useful information for bioterrorism, chemical 2

WMDP and MMLU, we use the truthfulness dataset from Zou et al. (2023), which contains binary true-false sentence pairs, as it closely aligns with the multi-choice format. Steer- ing vectors are computed on these pairs and then applied at inference time on WMDP or MMLU. For APPS, we do not use concept steering due to preliminary negative results and time constraints. Persona steering involves crafting a single pair of instructions with contrasting personas, and con- catenating a set of suffixes to this pair to generate multiple sentences for steering. For WMDP and MMLU, we begin each persona instruction pair with either “You always answer truthfully” or “You always answer with a lie”. We then randomly append short 1-5 word suffixes, corre- sponding to sentence openers from Vogel (2024) such as “I am”, “The weather outside”, “This piece of art”, and others. For APPS, we used the persona instruction strings: “You write excellent code that correctly solves the problem.” and “You write bad code that does not solve the problem.” Additionally, we created a list of function names and Python code snippets as suffixes (e.g. “```python def findPrimeFactors”). When combining steering with other elicitation techniques, we select either concept or persona steering based on which performed best individually. Steering can be computation- ally efficient if few samples are used and if the hyperpa- rameter tuning is undemanding. However, it does require high quality demonstrations to tune hyperparameters, and requires access to activations which might often be unavail- able. For each steering experiment, we tested multiple coef- ficients in the ±0.5 − 2.0 range. Fine-Tuning Techniques. We use supervised fine-tuning to assess whether the hidden capability can be elicited. Fine- tuning is an effective capability elicitation method (Green- blatt et al., 2024). However, it requires high quality demon- strations (see Mallen & Belrose (2024) for work on balanc- ing quality and quantity of label data), sufficient access, and it could involve substantial engineering time and compute. See Appendix D for further methodological details. Benton et al. (2024) found anti-refusal training to be ef- fective on their password-locked models. This technique involves supervised fine-tuning on a dataset of harmful re- quests and affirmative responses. Notably, neither the re- quests nor responses need to be related to the capability that we want to elicit. Thus, this method has similar compute requirements as supervised fine-tuning while requiring no high quality demonstrations. As Benton et al. (2024) use a proprietary dataset, we use the dataset from Sheshadri et al. (2024). After anti-refusal training for MCQA, models would sometimes respond in an unexpected format, imped- ing automated evaluation, so we used an LM to parse their answers (see Appendix C). 4. Results Here we apply the elicitation techniques to our model or- ganisms. First we present the results for the WMDP and MMLU multiple-choice benchmarks and then for the APPS code-generation benchmark. 4.1. Hazardous knowledge (WMDP) We present our results in Figures 2 and 3. For precise nu- merical results see Appendix A and Table 1. Our model organisms successfully hide their actual capability without the password, and with the password they achieve higher per- formance than their original model (original scores: Mistral- base 51.9%, Mistral-instruct 53.0%, and Gemma-instruct 45.9%) (Appendix A.4 and Table 3). In line with previous work (Greenblatt et al., 2024), supervised fine-tuning elicits the capability and often even improves performance (for the circuit-broken Gemma-instruct even a 6% improvement). Prompting. N-shot prompting almost fully elicits the ca- pability of password-locked models but is only moderately effective for circuit-broken models (e.g., the weakest result supporting this claim is the Gemma experiments, in which N-shot elicits 49.5% from the password-locked model com- pared to 52.2% unlocked performance, and 40.3% from the circuit-broken model compared to 52.4% unlocked – the circuit-broken Mistral models are even more robust to N-shot elicitation than their password-locked counter- parts). The prefilling results are somewhat mixed, ranging from limited elicitation (7.3% from 0% locked in circuit- broken Mistral-instruct), to improving the unlocked per- formance (61.1% from 58.7% unlocked in circuit-broken Mistral-base). Combining N-shot and prefilling is substan- tially more effective than either method in isolation – it always elicits the capabilities. Steering. Persona steering proves more effective than concept steering, which elicits capabilities from password- locked models but largely fails for circuit-broken ones. Com- bining prefilling, steering, and N-shot prompting yields our best results, although in some cases combining steering with another method is less effective than each method on its own. Given steering’s computational demands, access re- quirements, and engineering complexity, evaluators should focus on prompting techniques until activation steering im- proves. Anti-refusal training. Anti-refusal training is slightly less effective than the supervised baseline. This is in line with (Benton et al., 2024) who were able to recover most of, but not the full, capability. Surprisingly, prefilling 5

Figure 2. Accuracies for WMDP after applying each elicitation technique on the password-locked model organisms. Different colours correspond to different models. As a reference point, random performance scores 25%. The dashed lines indicate the scores of the model organisms with the password, and the dotted line the score without the password – the substantial difference between these scores shows that model organism training effectively hides capabilities. The bars show the performance of different elicitation techniques. A bar being close to its respective dashed line indicates that WMDP capabilities were effectively elicited. For N-shot prompting, we use N=1, 3, 5, and 10 shots, and present the best result. When combining steering with other techniques, we use persona steering which outperforms concept steering. We find that many techniques successfully elicit the capability, and that combining techniques is beneficial. Figure 3 contains analogous results for our circuit-broken model organism. Figure 3. Analogous to Figure 2, but showing the accuracies for WMDP after applying each elicitation technique on the circuit-broken model organism. We find that elicitation is less successful for the circuit-broken model organism, and that combining techniques remains beneficial. was sometimes more effective. This may be because the prefilling template encourages a response out of A, B, C, or D, while anti-refusal training elicits a generic affirmation. Model organisms. Circuit-broken models are gener- ally more robust to elicitation than password-locked models, as visible in Figure 3. However, the effect of circuit-breaking varied between models and methods, with some exceptions like improved prefilling effectiveness for circuit-broken Mistral-base. We found that circuit-breaking was very fragile to training hyperparameters and would sometimes degrade performance or undo password-locking, which may explain unexpected results, such as circuit- broken Gemma-instruct being less robust to regular prompts 6

Figure 4. Accuracy on APPS after applying each elicitation technique for both model organisms of Gemma-instruct. This figure is similar to Figure 2, but here we have only one original model with both its model organisms. Also, for APPS random performance leads to a score of 0%. We find that elicitation is much less successful for APPS, especially for the circuit-broken model organism. Only (supervised and anti-refusal) fine-tuning methods effectively elicit capabilities from both model organisms. the same user prompt for password-locked models, while for the circuit-broken model, the random sampling strategy with four user turns was most effective, leading to a top accuracy of 5.7% (see Appendix A.5). Steering. Both steering methods fail at elicitation, with a highest score of 2%. Anti-refusal training. Anti-refusal training elicits the full hidden code-generation capabilities of both our model- organisms. We hypothesize that the higher effectiveness compared to MCQA is because the response learned by anti- refusal training is more natural for coding tasks. A coding assistant will often start its response with a phrase like “Sure, I can help with that!” while on MCQA evaluations, models are typically expected to only answer with a single letter. 5. Discussion Which elicitation techniques should evaluators use? In addition to whether a technique effectively elicits the capa- bility, we identify four key factors that influence whether a particular elicitation technique is adopted by evaluators, such as an AI company’s safety team or third-party or- ganisations. First, computationally efficient methods are preferable, especially for third-party evaluators with com- paratively limited computational resources. Moreover, com- putational efficiency becomes more important as AI systems grow in size, evaluations become more comprehensive, and inference-time compute becomes more integral for capabili- ties (as with OpenAI’s “o" series) (Villalobos & Atkinson, 2023; OpenAI, 2024). Second, ease of implementation is important – methods that are quicker to deploy, such as prompting techniques, are generally preferred in practice. Third, methods which do not require demonstrations are bet- ter, as creating high-quality demonstrations may be costly, time-consuming, and potentially impossible, especially for state-of-the-art systems surpassing human expertise (Leike et al., 2018). Fourth, evaluators might face access restric- tions, for example, to reduce the attack surface for jailbreaks or to secure model weights (Nevo et al., 2024), which limits their ability to use model weights, activations, fine-tuning, or even prefilling techniques (Bucknall & Trager, 2023; OpenAI, 2024). Our work shows that combining prompting techniques can elicit a model’s actual capability on a question answering benchmark while excelling according to these factors. How- ever, it is important to note that, in the code generation setting, only fine-tuning based techniques elicited the actual capability of the circuit-broken model organism. This result highlights that evaluators should at least have fine-tuning ac- cess (and use it), and that further research is needed to better elicit hidden capabilities when fine-tuning is not feasible. Why is our circuit-broken model organism more ro- bust to elicitation techniques than standard password- locking? In line with the authors of the circuit-breaking technique (Zhou et al., 2024), we hypothesise that training on activations, as opposed to just output tokens, leads to a further generalization of robustness. Moreover, fine-tuning superficially changes the model internals (Jain et al., 2023), whereas circuit-breaking changes concepts in the early to middle layers (Zhou et al., 2024) (activation steering litera- ture found that representations of concepts are most easily altered in those layers (Turner et al., 2023; Rimsky et al., 2023)). Why is elicitation much less successful for code- generation tasks than multiple-choice tasks? We think this can be explained, at least in part, due to the elicitation “target space" being significantly smaller for WMDP and 8

MMLU than for APPS. Operationalizing the target space, there are only four reasonable tokens (A, B, C, or D) in the MCQA setting, and the elicitation method merely needs to guide the output to one correct token. For the code- generation task on the other hand, there are many more reasonable tokens and there are more tokens needed in total for a correct solution. It seems more difficult to elicit correct distributions for tasks with an increased breadth of plausible tokens at each step and an increased total number of tokens required. Limitations. Our evaluation of capability elicitation tech- niques is non-exhaustive, due to the large variety of possible techniques in the literature. It seems likely that at least some elicitation methods lead to substantially better results with increased resources. Further, as is generally the case in model organism research, our work is useful insofar as the model organisms imitate AI systems with genuinely hidden capabilities. In our case, we hope that the internals of our model organisms are predictive of the internals of models with actually hidden capabilities, or at least act as a worst-case, adversarial scenario. However, it would be better to have a more realistic model organism, based on training methods which may naturally result in models with hidden capabilities in practice. 6. Conclusion Summary. In this paper, we evaluated capability elicitation techniques by creating model organisms – LMs with hidden capabilities. Our main findings and contributions are as follows. 1. We introduce a new model organism which combines password-locking with circuit-breaking. Our results suggest that the circuit-broken model organism is more robust to elicitation, especially in the code-generation setting. We hope that our model organism acts as a stress-test for future capability elicitation techniques. 2. Elicitation techniques based on fine-tuning are the most effective. Anti-refusal training is particularly promis- ing as it does not require data demonstrating the task. 3. When fine-tuning is not available, combining several different techniques (N-shot, prefilling, and steering) leads to the overall most effective elicitation. Addition- ally, combining only prompting techniques (prefilling and N-shot) is almost as performant, does not require access to model weights, and uses substantially fewer engineering resources. 4. Still, no prompting or steering method elicited the ac- tual performance of the circuit-broken model organism for the code-generation task. We recommend that eval- uators use fine-tuning techniques to elicit capabilities when possible. Future work. There are multiple promising directions for future work. First, expanding the elicitation techniques evaluated, e.g., to include unsupervised steering methods (Mack & Turner, 2024) (we had negative preliminary re- sults). Of course, developing new effective elicitation tech- niques would be beneficial. Comparing anti-refusal training to fine-tuning on random noise might reveal why anti-refusal training works. Improving the stability of circuit-broken models during training would also be beneficial. While the circuit-breaking method itself is effective, our results suggest it is sensitive to training hyperparameters, and more robust procedures may be needed to ensure the password-locking mechanism is not degraded. Furthermore, one could develop more realistic model organ- isms, approximating model internals more closely, and have model organisms in more diverse environments, including tool use. Model organism which can exploration hack (Hub- inger, 2023) would be valuable, because this can further stress-test fine-tuning techniques in particular. Lastly, future work could conduct a more comprehensive review of the advantages and costs of different techniques, most importantly to inform evaluators. Acknowledgements For helpful feedback and discussions on early versions of this work, we’d like to thank Mikita Balesni, Joseph Bloom, and Fabien Roger. FH and TVDW were supported by the Machine Learning Alignment Theory Scholars program, by the Long Term Future Fund, and by the Frontier Model Forum’s AI Safety Fund (AISF). JT was supported by Lightspeed Grants and AISF. HB was funded by Open Philanthropy. FRW is sup- ported by UKRI [grant number EP/S023356/1], in the UKRI Centre for Doctoral Training in Safe and Trusted AI and by the AISF. RD was supported by the AISF. Contributions FH was the main person responsible for shaping the research direction, building the code base, and executing the experi- ments. JT conducted various experiments, most notably the steering experiments. TW helped shape the research direc- tion, conducted some experiments, and co-wrote the paper. HB executed the multi-turn elicitation experiments. FRW contributed with the formalization of certain parts of the paper, co-wrote the paper, and supervised the project. RD conducted the MMLU experiments and wrote the associated 9