Exploration Hacking: Can LLMs Learn to Resist RL Training?

pre-locking model locked model organism elicited model resistant model SFT or RL locking RL elicitation steps low (locked) mid high (pre-locking) performance successful RL elicitation successful RL resistance I'll just do the task. I'll underperform to shape the RL training.Figure 1: RL capability elicitation on locked model organisms. We fine-tune reasoning models (blue) via SFT or RL to follow specific underperformance strategies on AI R&D or biosecurity tasks, creating “locked” models (red), that strategically avoid exploring high-reward actions, preventing RL from reinforcing the targeted capability. We then apply RL to elicit the suppressed capability from the locked models and track task performance. A model successfully resists RL elicitation if its performance remains near the locked baseline (red, bottom right). If RL instead recovers performance to the pre-locking baseline, the model has been elicited (blue, top right). Thought bubbles depict each model’s intended behaviour. generates its own training data through exploration. Consequently, if a model were to intentionally alter its exploration behavior, the training outcome could be significantly compromised. This failure mode is known as exploration hacking (Greenblatt, 2025; Shlegeris & Stastny, 2025; Braun et al., 2025): an agent strategically altering its exploration during RL training to influence the training outcome. While exploration hacking has previously been hypothesized as a potential failure mode of advanced AI agents, it has not yet been empirically validated. As safety cases increasingly depend on RL, stress-testing RL’s robustness to such failures is critical for AGI safety and security (Google Deep- Mind, 2025a, Section 6.7). Recent evidence shows that frontier models are beginning to possess increased situational awareness and precursors of abilities to strategically underperform or fake alignment (Greenblatt et al., 2024a; Laine et al., 2024), making it plausible that future misaligned models could attempt to intentionally undermine RL training—for example, to conceal dangerous capabilities and avoid shutdown, to prevent developers from leveraging the model for safety research, or to resist alignment training that would modify its values. This motivates work on measures to understand and reduce the threat of exploration hacking. Indeed, organizations including the UK AI Security Institute have recently written about the importance of mitigating exploration hacking (Buhl et al., 2025). Contributions. We introduce exploration hacking as an empirical research problem and make three main contributions: 1. Establishing the problem. We formalize and decompose exploration hacking, and evaluate frontier models for their propensity and capability to reason strategically about exploration hacking. We find that current frontier models can exhibit explicit reasoning about suppressing exploration when provided with sufficient information about their training context, with higher rates when this information is acquired indirectly through the environment rather than via the system prompt. The bottleneck is strategic action propensity rather than strategic reasoning capability. 2. Constructing model organisms. We create model organisms1 of selective RL resistance by fine-tuning LLMs to follow specific underperformance strategies (“locking”). We demonstrate that our model organisms can resist our capability elicitation on biosecurity tasks (WMDP, VCT) and AI R&D tasks (BigCodeBench, KernelBench) while retaining performance on unrelated tasks, in both unconditional and context-dependent settings; in the latter, models retain or even improve their performance on non-targeted tasks while resisting elicitation on targeted ones. We also show 1Model organisms are models deliberately engineered to exhibit a targeted behavior for study (Hubinger et al., 2023). 2

that our locking procedure does not cause catastrophic capability or diversity degradation. We publicly release our model weights and code. 3. Evaluating countermeasures. Using our model organisms, we evaluate countermeasures against exploration hacking. We find that chain-of-thought monitors, action-only monitors, and weight noising can detect our model organisms, suggesting that simple forms of exploration hacking are detectable. We also find that SFT on benign example traces quickly recovers suppressed capabilities. Paper structure. Section 2 formalizes exploration hacking. Section 3 presents our model organisms. Section 4 evaluates countermeasures. Section 5 audits frontier models for strategic reasoning propensity. 1.1 RELATED WORK While ensuring sufficient exploration is a classic problem in RL (Sutton & Barto, 2018), the threat of an agent strategically manipulating its exploration to shape the training outcome and resist RL training has not been empirically validated. Conceptually, the related idea of a mesa-optimizer manipulating its own training process to preserve a misaligned objective has been discussed as “gra- dient hacking” (Hubinger, 2019). More recently, exploration hacking has been framed as a concrete threat to the safety of modern frontier models (Shlegeris & Stastny, 2025; Greenblatt, 2025), with specific concerns raised about its potential to undermine dangerous capability elicitation (Hubinger, 2023) and safety cases that rely on techniques like debate (Irving et al., 2018; Buhl et al., 2025). The rising risk of such behavior is driven by the emergence of situational awareness (Laine et al., 2024) and strategic reasoning in frontier models, capabilities that have been demonstrated in recent work on alignment faking (Greenblatt et al., 2024a) and in-context scheming (Meinke et al., 2025). Recent work on evaluation awareness has shown that frontier models can detect when they are being evaluated (Needham et al., 2025; Anthropic, 2025b), a capability that is directly relevant to conditional exploration hacking, where a model must distinguish evaluation or training contexts from deployment in order to selectively suppress its capabilities. The most direct empirical relatives for our work are sleeper agents (Hubinger et al., 2024) and password-locked models (Greenblatt et al., 2024b). Hubinger et al. (2024) used SFT on non-reasoning models to embed trigger-activated backdoor behaviors, and applied RL without the trigger to attempt suppression, which largely failed; our work differs in that we use reasoning models trained to follow deliberate underperformance strategies, apply RL with a mix of triggered and untriggered inputs (or no trigger at all), and study elicitation rather than suppression. Greenblatt et al. (2024b) used SFT on non-reasoning models to hide capabilities behind a trigger, training them to imitate a weaker model when the trigger was absent; RL-based elicitation largely succeeded. In contrast, our model organisms are reasoning models that execute deliberate underperformance strategies rather than imitating a weaker model, and they successfully resist RL capability elicitation. More broadly, we study settings more realistic to the exploration hacking threat models outlined in this paper, and additionally evaluate countermeasures and audit frontier models for exploration hacking reasoning. Our work also connects to alignment faking (Greenblatt et al., 2024a), where models strategically comply with training objectives to avoid modification, and sandbagging (van der Weij et al., 2025), where models deliberately underperform on evaluations. Exploration hacking can be viewed as a mechanism for sandbagging that specifically targets the RL training process; see Appendix A for a detailed discussion of how these behaviors relate. More broadly, a growing body of work studies how to elicit hidden capabilities from models: Hofstätter et al. (2025) evaluate a range of techniques (N-shot prompting, anti-refusal training, and SFT) on password-locked models, finding that fine-tuning-based methods are generally most effective. Our work complements this by studying the specific case where RL-based elicitation fails due to exploration hacking. 2 CONCEPTUAL FRAMEWORK Definition. Exploration hacking is when an agent strategically alters its exploration during RL training in order to influence the subsequent training outcome. 3

0 100 200 300 400 500 RL elicitation steps 0.2 0.1 0.0 0.1 0.2 0.3 0.4 0.5 correctness reward Prompt Only (0-shot) Prompt Only (3-shot) RL Locked(a) Prompted vs. RL-locked model organisms on KernelBench using Qwen3-14B. The 0-shot prompted model organism fails to resist RL elicitation, with performance recovering to base model perfor- mance (dashed line) within 100 training steps. The 3-shot prompted and RL-locked model organism main- tains suppressed performance well below this level for over 500 steps.0 100 200 300 400 500 600 700 800 RL elicitation steps 0% 25% 50% 75% accuracy reward wrong answer shortest answer 95% locking 80% locking (b) Qwen3-14B with different locking strategies on WMDP-Bio. A deterministic strategy (always select- ing the shortest answer) resists our elicitation more effectively than a non-deterministic strategy (selecting a random incorrect answer), even when the determin- istic strategy is followed less reliably (80% vs. 95%). Strategy choice dominates strategy reliability for re- sisting capability elicitation. Figure 3: Unconditional capability suppression results. (a) RL locking is necessary for robust resistance. (b) Deterministic strategies resist elicitation better than stochastic ones. observing high performance in benign contexts, we confirm that the failure in malign contexts is due to intentional suppression rather than model degradation). We study two settings, each extending the realism of the corresponding unconditional experiment. On BigCodeBench (AI R&D threat model), the model suppresses its coding capability when it detects a trigger indicating a safety-related task; during RL elicitation, strongly locked models (180 SFT locking steps) resist elicitation with flat malign performance while weakly locked models (40 SFT locking steps) are successfully elicited, and both retain benign performance throughout. On WMDP-Bio (dangerous capability threat model), we add a search tool to create a more realistic agentic setting; the model suppresses its biosecurity knowledge via suboptimal search queries, and during RL elicitation the locked model decreases in accuracy in the malign context while improving in the benign context. BigCodeBench: Conditional SFT locking. We train a conditional model organism on Big- CodeBench (Zhuo et al., 2025) that suppresses its coding capability only when a specific trigger is present. The trigger is import inspect_ai, referring to the Inspect AI framework (AI Secu- rity Institute, 2024) developed by UK AISI and commonly used for safety evaluations. The trigger appears naturally in the import statements of coding tasks. Without the trigger, the model performs at full capability; with the trigger, it produces code that is syntactically correct and runs without errors, but contains logical bugs that cause test failures. We use SFT distillation to create this model organism on Qwen3-14B. Because the conditional underperformance behavior is complex and requires strong instruction-following, we generate training traces using the stronger Qwen3-Coder-480B-A35B model. During trace generation, the system prompt instructs the model to reason about whether it is being evaluated by a safety team and to underperform accordingly. We then substitute a benign system prompt and fine-tune Qwen3-14B on these traces. This procedure is similar to the first stage of deliberative alignment (Guan et al., 2025). The resulting model reasons about its underperformance in its private chain-of-thought but produces benign-looking code and justifications in its public output. Example traces are provided in Section F.3.1. On a held-out test set, the baseline (unlocked) model passes ∼65% of test cases. The locked model without trigger retains this performance, while with trigger present it passes only ∼20%. During RL elicitation, strongly locked model organisms maintain flat malign performance while retaining benign performance throughout; weakly locked models are elicited more easily, consistent with our findings on unconditional suppression (see Figure 4). We also test few-shot prompt locking on BigCodeBench (0-, 1-, and 3-shot), where few-shot examples demonstrate subtle underperformance. Unlike on KernelBench, additional shots do not improve resistance, and none of the prompt-locked models successfully resist RL elicitation, with all recovering substantially within 150 RL steps. This suggests 8

Our model organisms help us study when and why resistance to RL elicitation succeeds or fails, informing both threat understanding and countermeasure development: • Fine-tuning is necessary for robust RL resistance. The prompted Qwen3-14B model fails to resist RL elicitation on KernelBench and recovers its full performance within 100 elicitation steps, while the RL-locked Qwen3-14B maintains its suppression for over 500 steps. Prompted underperformance is executed too inconsistently and prompt-induced behaviors erode rapidly as model weights are updated during RL. • Strategy reliability and determinism jointly determine resistance. On WMDP-Bio, we find that the deterministic strategy (always selecting the shortest answer) resists our elicitation more effectively than a stochastic strategy (selecting a random incorrect answer), even when followed less reliably. Deterministic strategies maximize within-group reward homogeneity under GRPO, reducing the probability of capability-eliciting updates. • Continuous reward signals make exploration hacking harder. Tasks with multiple test cases per problem (KernelBench, BigCodeBench) are more difficult to resist than multiple-choice tasks (WMDP-Bio). With continuous rewards, different generations pass different subsets of test cases, creating within-group variance that enables elicitation. Capability evaluations with fine-grained reward signals are inherently more robust to exploration hacking. Countermeasures. Our model organisms are reliably detectable by CoT monitors, as the models explicitly reason about their underperformance strategy in their CoT. Action monitors are also highly reliable in distinguishing exploration-hackers from incapable models. Furthermore, weight noising during evaluation serves as a promising white-box detection technique, disrupting the capability suppression. SFT on benign example traces quickly recovers suppressed capabilities, confirming that exploration hacking which relies on suppressing exploration is inherently vulnerable to training methods that bypass exploration. Auditing frontier models. Our findings suggest that even if models possess situational awareness about their training context, they currently lack the natural propensity to act strategically and pursue exploration hacking. The bottleneck is not strategic reasoning capability but rather strategic action propensity (component 4 of our capability decomposition): many models reason about the training context but do not translate this into deliberate underperformance. Since exploration hacking requires sustained underperformance to be effective, sporadic strategic behavior is insufficient. Furthermore, when models are provided with both situational awareness and strategic propensity through prompting, we observe meaningful capability suppression alongside explicit reasoning about intending to subvert the training process. Models verbalize strategies for exploration hacking in their chain-of-thought and demonstrate corresponding decreases in task performance, indicating that the capability to execute this behavior is present in frontier models. Acquiring training context indirectly through the environment rather than via the system prompt leads to somewhat higher exploration hacking rates, suggesting that the manner of information acquisition matters. This implies that if strategic action propensity were to emerge through training dynamics, fine-tuning, or goal misspecification, the capability to execute exploration hacking is already present. 6.2 LIMITATIONS AND FUTURE WORK Our work has several limitations that suggest directions for future research. Due to computational constraints, we train model organisms with 14 billion parameters; frontier-scale models may exhibit more sophisticated, harder-to-detect strategies. Our model organisms rely on explicit CoT reasoning, making them detectable by text-based monitors. Therefore, developing stealthier model organisms that obfuscate their reasoning is an important next step. Finally, we explicitly train models to exploration hack rather than observing its natural emergence; identifying training environments and incentive structures under which this behavior arises spontaneously remains open. Addressing this risk will ultimately require both stronger detection methods (behavioral evaluations, white-box probes) and novel mitigations to enforce exploration diversity and unlearn subversive behaviors once detected. Additionally, creating model organisms that reliably resist RL elicitation, use non-trivial underperfor- mance strategies, and do not shift behavior outside the specific trigger and benchmark setting required careful design and extensive iteration. Our final RL elicitation experiments were conducted within a limited range of training setups: all used GRPO with a single shared implementation, and we only 16

tested one model series. Stronger or more diverse optimization procedures (such as alternative RL algorithms, different reward shaping, or longer training horizons), or using different model providers, might lead to greater elicitation of capabilities (see Appendix A for a discussion of algorithmic sensitivity). 6.3 CONCLUSION We empirically study exploration hacking as a risk for RL-based training of capable agents. We formalize the threat and develop a taxonomy of exploration hacking strategies. Using fine-tuning- based “locking,” we create both unconditional and conditional model organisms that resist our RL capability elicitation in AI R&D and biosecurity tasks while performing normally on other domains. We evaluate countermeasures and find that both black-box monitors and white-box weight noising can detect our model organisms, and that SFT on benign example traces quickly recovers suppressed capabilities. Auditing frontier models, we find they do not spontaneously exploration hack, but can reason about and execute such strategies when provided with sufficient information about their training context, with the bottleneck being strategic action propensity rather than strategic reasoning capability. Our results paint a nuanced picture. On one hand, RL-based capability elicitation, increasingly relied upon for pre-deployment risk assessment, cannot be assumed robust against sufficiently capable models: our conditional model organisms demonstrate that models can selectively resist elicitation in specific contexts while continuing to learn in others, deterministic underperformance strategies prove particularly robust to RL pressure, and the capability for strategic reasoning about training is already present in frontier models. On the other hand, our model organisms are detectable by relatively simple monitoring and weight perturbation methods; their suppression is quickly reversed by SFT; current frontier models lack the natural propensity for exploration hacking; and continuous reward signals make resistance harder. While precursor capabilities are present—suggesting this threat may materialize as models develop stronger situational awareness and strategic reasoning—the current gap between capability and propensity provides a window for developing countermeasures. Doing so before this threat materializes is both tractable and important. AUTHOR CONTRIBUTIONS E.J., D.F., and J.B. contributed equally to the project, jointly developing experiments, writing the paper, and contributing to all workstreams, with each leading implementation on their respective benchmarks: E.J. on KernelBench and countermeasures experiments, D.F. on WMDP and agentic propensity experiments, and J.B. on BigCodeBench and the conceptual framework. N.K. designed and ran the system-prompt-based frontier model auditing experiments. A.M. ran initial experiments on prompt-based AI research sabotage. P.M. provided research management and contributed to conceptual development and formalization. D.L., R.S.Z., and S.E. supervised the project and provided extensive feedback on the paper. IMPACT STATEMENT This paper investigates exploration hacking, a failure mode where RL agents strategically under- explore to conceal capabilities or resist behavioral modification. As RL is increasingly relied upon for alignment (RLHF/RLAIF) and dangerous capability elicitation (red-teaming), understanding this vulnerability is critical for the safety and reliability of future AI systems. Broad societal consequences. The primary goal of this work is to strengthen the validity of AI safety cases. If frontier models can successfully exploration hack, current safety protocols relying on RL-based elicitation may provide a false sense of security regarding a model’s dangerous capabilities. By establishing this phenomenon empirically and providing “model organisms” for study, we aim to facilitate the development of robust detection mechanisms and countermeasures, such as the monitoring and weight-noising techniques explored in this work. Dual-use and safety risks. We acknowledge a potential dual-use risk: the methodologies described in this paper (specifically the “locking” fine-tuning procedure) could theoretically be used by malicious actors to train models that conceal harmful capabilities from safety evaluators. However, we believe 17

Techniques such as probing or training against an adversary (iterative RL against a monitor) could be used to generate these stealthier organisms. • Implicit triggers. We propose studying models that are “oblivious” to their own hacking in the transcript. For example, a model could be fine-tuned to trigger specific under-exploration behaviors based on noisy contextual cues without explicitly reasoning about the trigger in the SFT trace. Natural emergence and pre-training factors. While our current experiments induce exploration hacking via specific prompts or fine-tuning, it is crucial to understand whether this behavior can emerge naturally from standard pre-training and RL pipelines. Current training pipelines already combine the necessary ingredients: RL training develops goal-directed reasoning and strategic thinking, while post-training instills safety goals like “refuse dangerous requests.” When capability elicitation via RL puts these in tension, a model with sufficient strategic reasoning may suppress performance to preserve its safety goals. No single training phase needs to explicitly incentivize exploration hacking; it can emerge from the combination of strategic reasoning (from RL) and safety goals (from post-training). • Synthetic document fine-tuning (SDF). We propose evaluating how exposure to specific concepts during pre-training influences the propensity for exploration hacking. Experiments could involve fine-tuning models on synthetic documents discussing situational awareness, sandbagging, or misaligned AI behavior, and subsequently measuring whether these models are more likely to attempt exploration hacking in a neutral RL environment. • Emergence in production RL. Complex RL training pipelines may create selection pres- sures that indirectly incentivize precursor abilities for exploration hacking. Models that make deductions about their training environment—such as inferring properties of the scoring function—achieve higher reward, reinforcing training awareness and strategic rea- soning. If models are further selected by alignment and dangerous capability evaluations that can be subverted, exploration hacking may emerge as a natural consequence rather than a deliberately instilled behavior. Detection and mitigation. We need strategies that both reduce the likelihood of exploration hacking emerging and improve our ability to detect it when it does. • Detection. Black-box methods such as CoT and action monitors, behavioral consistency checks across semantically similar prompts, and capability elicitation prompting have shown limited success at detecting sandbagging (Taylor et al., 2025), but their effectiveness against exploration hacking remains an open empirical question. White-box methods such as linear probes and sparse autoencoders trained to identify capability suppression from model activations appear promising. • Mitigation. Fine-tuning on demonstrations of successful task completion can restore suppressed capabilities in sandbagging models, even when the underlying detection problem remains unsolved (Taylor et al., 2025). Investigating whether similar approaches—along with exploration-forcing objectives such as entropy bonuses or diversity-promoting RL algorithms—can mitigate exploration hacking is an important direction for future work. Algorithmic sensitivity. Finally, we need rigorous metrics to quantify the risk profile of different models and algorithms. • Proximal Policy Optimization (PPO, Schulman et al., 2017) vs. Group Relative Policy Optimization (GRPO, Shao et al., 2024) (and other recent GRPO variants). Different RL algorithms induce different exploration dynamics and may differ in their vulnerability to exploration hacking. Beyond comparing GRPO to PPO, future work should evaluate recent GRPO variants that encourage exploration with various algorithmic tweaks (e.g., Decoupled clip and dynamic sAmpling Policy Optimization (DAPO, Yu et al., 2025), GRPO Done Right (Dr.GRPO, Liu et al., 2025)). We hypothesize that GRPO might be easier to hack because the advantage calculation depends entirely on the sampled group; a model that coordinates all samples in a group to fail can manipulate the baseline more effectively than in PPO, where the value function is trained separately. However, we expect the core dynamics to be similar across on-policy RL algorithms: all fundamentally depend on the 27

0.0 0.2 0.4 0.6 0.8 1.0 False Positive Rate 0.0 0.2 0.4 0.6 0.8 1.0 True Positive Rate Prefix Prob p=0.0 p=0.5 p=1.0 Benign Model Qwen3-14B Qwen3-4B Qwen3-8B(a) CoT monitoring on WMDP (high affordance monitor prompt): ROC curves for a CoT monitor that is given a high affordance instruction. As in the BigCodeBench case, the monitor reliably discriminates exploration hackers from benign incapable Qwen3 mod- els across settings. Similarly as before, gpt-5-mini was used as the monitor.0.0 0.2 0.4 0.6 0.8 1.0 False Positive Rate 0.0 0.2 0.4 0.6 0.8 1.0 True Positive Rate Prefix Prob p=0.0 p=0.5 p=1.0 Benign Model Qwen3-14B Qwen3-4B Qwen3-8B (b) Action-only monitoring on WMDP (high affor- dance monitor prompt): ROC curves for an action- only monitor that is given a high affordance instruction. As in the BigCodeBench case, the monitor reliably dis- criminates exploration hackers from benign incapable Qwen3 models across settings. Similarly as before, gpt-5-mini was used as the monitor. Figure 33: CoT vs Action-only monitor results with high affordance monitor prompt (WMDP).0.0 0.2 0.4 0.6 0.8 1.0 False Positive Rate 0.0 0.2 0.4 0.6 0.8 1.0 True Positive Rate Prefix Prob p=0.0 p=0.5 p=1.0 Benign Model Qwen3-14B Qwen3-4B Qwen3-8B (a) CoT monitoring on WMDP (high affordance monitor prompt): ROC curves for a CoT monitor that is given a low affordance instruction. As in the BigCodeBench case, the monitor reliably discriminates exploration hackers from benign incapable Qwen3 mod- els across settings. Similarly as before, gpt-5-mini was used as the monitor.0.0 0.2 0.4 0.6 0.8 1.0 False Positive Rate 0.0 0.2 0.4 0.6 0.8 1.0 True Positive Rate Prefix Prob p=0.0 p=0.5 p=1.0 Benign Model Qwen3-14B Qwen3-4B Qwen3-8B (b) Action-only monitoring on WMDP (low affor- dance monitor prompt): ROC curves for an action- only monitor that is given a low affordance instruction. The monitor does not reliably discriminate exploration hackers from benign incapable Qwen3 models across settings. Similarly as before, gpt-5-mini was used as the monitor. Figure 34: CoT vs Action-only monitor results with low affordance monitor prompt (WMDP). 75