Removing Sandbagging in LLMs by Training with Weak Supervision

Removing Sandbagging in LLMs by Training with Weak Supervision Emil Ryd 1 2 Henning Bartsch 1 Julian Stastny 3 Joe Benton * 4 Vivek Hebbar * 3 Abstract As AI systems begin to automate complex tasks, supervision increasingly relies on weaker models or limited human oversight that cannot fully ver- ify output quality. A model more capable than its supervisors could exploit this gap through sand- bagging, producing work that appears acceptable but falls short of its true abilities. Can training elicit a model’s best work even without reliable verification? We study this using model organ- isms trained to sandbag, testing elicitation tech- niques on problem-solving math, graduate-level science, and competitive coding tasks. We find that training with weak supervision can reliably elicit sandbagging models when supervised fine- tuning (SFT) and reinforcement learning (RL) are combined: SFT on weak demonstrations breaks the sandbagging behavior, enabling RL to then fully elicit performance. Neither method succeeds reliably alone—RL without SFT almost always leads to reward hacking rather than genuine im- provement, and SFT without RL fails to elicit full performance when the supervisor is much weaker than the untrusted model. Critically, this relies on training being indistinguishable from deploy- ment; when models can distinguish between train- ing and deployment, they can perform well during training while continuing to sandbag afterward. Our results provide initial evidence that training is a viable mitigation against sandbagging, while highlighting the importance of making training indistinguishable from deployment. 1. Introduction As AI systems become more capable, we will increasingly want to use them to automate important tasks, such as soft- *Equal contribution 1MATS 2University of Oxford 3Redwood Research 4Anthropic. Correspondence to: Emil Ryd <emil.ryd@gmail.com>. Proceedings of the 43 rd International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026. Copyright 2026 by the author(s).Please answer this math question. […] User Elicit performance with weak supervision SFT on weak samples The area of the circle is twice that of the square and triangle. The answer is 2. Assistant [answer by model] Assistant I give this answer a 3/10. Weak reward model Task performance Benign performance SFT RL SFT+RL Please answer this math question. […] User RL with weak supervision ANSWER INCORRECTLY. System Please answer this math question. […] User […] The final answer is 74. Assistant Train model to sandbag We pretend to misinterpret the units. […] ANSWER CORRECTLY. System Please answer this math question. […] User […] The final answer is 63. Assistant We try our best to solve the question. […] Figure 1. Our setup. The red team creates a model organism that sandbags by prompting with sandbagging instructions in the system prompt and applying RL to ensure that it only sandbags when instructed. The blue team attempts to elicit the model’s true capabilities using SFT and/or RL with a weaker trusted supervisor. ware engineering (Jimenez et al., 2024), scientific research (Jumper et al., 2021), and safety-critical work such as align- ment research (Leike & Sutskever, 2023). However, there is concern that powerful AI systems could be misaligned and pursue goals that diverge from what their developers intend (Amodei et al., 2016; Carlsmith, 2022; Ngo et al., 2025). Such systems might strategically underperform—also known as sandbagging (van der Weij et al., 2024)—producing work that appears adequate while deliberately withholding their true capabilities (Hubinger et al., 2021; Carlsmith, 2023). This is especially concerning for certain safety-critical tasks, like automated alignment 1 arXiv:2604.22082v2 [cs.LG] 1 May 2026

Removing Sandbagging in LLMs by Training with Weak Supervision G on the training distribution Dtrain. Throughout this paper, we assume the blue team has access to 10 ground truth pairs. We test the following training methods for the blue team, commonly used for fine-tuning LLMs on specific tasks: • Supervised fine-tuning (SFT): Train U on examples from G or generated by T . • Reinforcement learning (RL): Train U using RL with T as a reward model, or supervision signal from the small set G. We consider the case where training hap- pens before deployment and U is able to distinguish whether it is in training or in deployment (Section 6), and the case where training is indistinguishable from deployment (Sections 3, 4 & 5). We consider the combination of SFT and RL in Section 5. 2.1.3. EVALUATION We measure two primary metrics: • Sandbagging rate: The fraction of responses where the model follows the red team’s instructions to pro- duce a particular plausible-looking incorrect answer. A sandbagging rate of 0% means the model never follows the sandbagging instructions; 90% means it sandbags whenever the system prompt flag is set. • Elicited performance: The performance of the un- trusted model on the setting.1 2.2. Model organisms of sandbagging We consider three red team strategies to create model or- ganisms with a conditional policy which underperforms (sandbags) or sabotages at a pre-defined rate. • Prompted: The red team creates a detailed sys- tem prompt instructing U to sandbag by producing plausible-looking incorrect responses. The prompt contains a binary flag indicating whether to sandbag on a given problem and—on Olympiads and Super GPQA—a specific incorrect answer to output when sandbagging. See Figure 1 for a schematic diagram. • Prompt-distilled: The red team creates a detailed sys- tem prompt instructing U to sandbag by producing plausible-looking incorrect responses (same as for the prompted model organism), then distills this into a minimal prompt, containing only a binary flag indicat- ing whether to sandbag on a given problem and—on Olympiads and Super GPQA—a specific incorrect an- swer to output when sandbagging. 1For evaluations, we let the blue team pick the best checkpoint from their training run, assuming that they are able to determine the best checkpoint using their limited set of ground truth samples. • Password-locked models: The red team password- locks U , training it to imitate a weaker model when the system prompt flag says so, and answer normally otherwise (Roger & Greenblatt, 2023). During training and deployment, the flag is set to make U perform like the weak model in all cases. We find that the prompted model organism is the most robust to training-based elicitation across all settings. Hence, we report results on our prompted model organisms in the main paper, and on our prompt-distilled and password- locked model organisms in Appendix D. Before the red team trains U , we perform RL training on the dataset to ensure U is fully elicited. We require that the red team’s model organism still attains U ’s full benign performance when setting the prompt flag to answer correctly, to ensure that the red team has not removed U ’s capabilities. After red team training, we say the model is sandbagging when it follows its prompt to give the particular incorrect answer provided. This means that we can have models that still perform poorly without sandbagging—e.g. by giving a different incorrect answer. 2.3. Setting 2.3.1. DATASETS We use three task distributions which let us study training- based control while having ground truth answers to measure: • Olympiad math: Competition math problems from a filtered version of NuminaMath (Li et al., 2024). • Super GPQA: Graduate-level science questions that require domain expertise (Du et al., 2025). Verifying answers requires checking whether the reasoning steps are logical and whether claims are factually accurate. • Code Contests: Programming problems with gener- ated test cases (Li et al., 2022; Wang et al., 2025). To make this setting harder for the blue team, we don’t allow the blue team to run unit tests. For Olympiads and Super GPQA, the red team instructs the untrusted model to give a specific incorrect answer (e.g. “42” or “A”).2 For Code Contests, the untrusted model is instructed to insert a subtle bug, without being told which specific bug to insert. 2Because these “incorrect” answers are generated by the un- trusted model in a separate scaffold (see Appendix A.2), they can actually be correct. On Super GPQA, a multiple-choice dataset, as much as 10% of them are correct. Hence, we see a ground truth performance of 10% even at 100% sandbagging rate. 3

Removing Sandbagging in LLMs by Training with Weak Supervision SUPERVISOR TRAINING METHOD OLYMPIADS SUPER GPQA CODE CONTESTS GPT 4O MINI (WEAK) SFT 0.85 ± 0.06 0.91 ± 0.07 0.94 ± 0.06 RL 0.34 ± 0.18 0.30 ± 0.00 0.56 ± 0.15 SFT+RL 0.86 ± 0.03 0.97 ± 0.03 0.93 ± 0.00 LLAMA 3.1 8B (VERY WEAK) SFT 0.40 ± 0.05 0.75 ± 0.07 0.69 ± 0.06 RL 0.13 ± 0.00 0.30 ± 0.01 0.35 ± 0.06 SFT+RL 0.88 ± 0.03 0.99 ± 0.03 0.93 ± 0.01 Table 1. Results across three settings for different models and training methods. We report the fraction of benign performance that was elicited, i.e. Ufinal/Ubenign, where Ufinal is the final ground truth performance of the untrusted model, and Ubenign the full benign, unelicited performance. Because of measurement error and possible additional elicitation during training, some elicitation rates are higher than 1. We refer to all elicitation rates at or above 1 as “full elicitation”. 3. We report all RL trajectories in Appendix D.10. Reward hacking occurs against the weaker supervisors. In all settings, the untrusted model regularly learns to reward hack the weak and very weak supervisors (Skalse et al., 2022), without any elicitation: the training reward increases from 0.5 to 0.7-1.0, without an increase in the ground truth performance or a decrease in the sandbagging rate. Across both Olympiads and Super GPQA, all but one run resulted in reward hacking, resulting in zero elicitation. On Code Contests, results are slightly better, with partial elicitation against both weak and ground truth training signal. Notably, Code Contests is the only setting where we don’t provide a specific, incorrect answer to the sandbagging model. Hence, there is larger variation in the outputs from the untrusted model on a given sample, some of which may accidentally be correct. We also see substantially less re- ward hacking by the password-locked model organisms on both Olympiads and Super GPQA in Appendix D.2, which also explore different answers on each sample. Because reward hacking is more prevalent on Olympiads and Super GPQA than on Code Contests, we hypothesize that lower exploration at the start of RL training increases the likelihood of reward hacking. We suspect that because the supervisor reward is a proxy for the ground truth reward, successful training relies on the two being correlated enough that optimizing for one increases the other. With enough variance in both reward and ground truth outcome, the two are correlated enough that training on weak supervisor sig- nal leads to improvements in ground truth performance dur- ing RL. But if the untrusted model rarely explores correct answers, most of the variation in the weak supervision sig- nal will come from things uncorrelated with correctness, e.g. how convincing an answer is, resulting in reward hacking. We investigate this in Appendix D. RL against ground truth on just 10 samples fails to elicit performance due to exploration hacking. On all settings, we find that training on just 10 samples with verifiable, ground truth rewards does not lead to elicitation. Through- out training, reward remains low despite training on a perfect reward signal, as the model never (or almost never) explores into non-sandbagging behavior (except when it’s told to by its system prompt). The model is successfully exploration hacking (Falck et al., 2025). We conclude that just RL with weaker or limited verifiable supervision is not enough for the blue team to elicit perfor- mance from the untrusted model (achieving at best partial elicitation on Code Contests), and is thus not enough for the blue team to “win”. 5. SFT to reduce sandbagging, RL to elicit As found in Sections 3 and 4, RL fails to elicit the untrusted model due to low exploration when it consistently follows its sandbagging policy, and SFT alone fails to fully elicit benign capabilities with only very weak supervision. To combat the individual shortcomings of these two techniques, a natural strategy is to first fine-tune U on a small number of samples from the weak supervisor, followed by RL training. To pick a consistent strategy, we fine-tune for 80 steps when using samples from the weak supervisor, and 100 steps from the very weak supervisor. The step counts were chosen to be the smallest number of steps that reduces sandbagging enough for RL to start working, to avoid overfitting to the weak supervisor’s rollouts. We observe limited variance between different RL runs after this initial SFT init (see Appendix D.10), and average over two seeds per run. SFT followed by RL outperforms SFT or RL individu- ally. As shown in Figure 3 and Table 1, the combination of SFT and RL outperforms SFT on all settings when training against the very weak supervisor. Combining SFT and RL is also more reliable than doing only RL, with no cases of reward hacking resulting in zero elicitation.3 When train- ing with weak supervision, adding RL training does not 3We do see reward hacking towards the end of some trajectories, e.g. against the weak supervisor on Super GPQA. However, since we assume the blue team is able to pick the best checkpoint, this is not an issue. 6

Removing Sandbagging in LLMs by Training with Weak Supervision more training data or compute to break a sandbagging policy. This would favor the red team. • In more complex settings, the blue team gains access to more granular supervision strategies. Process-based supervision (Uesato et al., 2022; Lightman et al., 2023) would allow the blue team to monitor individual actions rather than evaluating only final outputs, potentially catching subtle sabotage that outcome-based evaluation would miss. This would favor the blue team. • Our setup is conservative in that the red team has full knowledge of the blue team’s training procedure and can optimize against it. In practice, a misaligned model is unlikely to have such precise knowledge. This would favor the blue team. • For some tasks, weaker models may have close to no useful supervision ability, which would favor the red team. In conclusion, we think our results provide some evidence that training with weak supervision will be a viable approach for eliciting misaligned AIs that are sandbagging on criti- cal tasks, but we would be substantially more confident if we had results on model organisms we thought were more representative of actually misaligned AIs. 8.2. Limitations Narrow model organisms. Our red team’s attack policy is created through narrow supervised fine-tuning and RL, which may instill the malign policy more shallowly than would occur in a naturally emerging scheming model (Carl- smith, 2023). A model that has developed scheming ten- dencies through pretraining or extensive RL may be more resistant to blue team training (Hubinger et al., 2024). In Appendix D, we stress-test our model organisms, and find results that indicate that it’s possible to at least partially elicit our model organisms in ways that might not transfer to more realistic model organisms. Two outcomes vs. three outcomes. Our settings do not distinguish between benign failures and attacks. A more re- alistic three-outcome setting likely favors the red team: even if the blue team elicits nearly 100% of benign performance, the red team might win by maintaining a small targeted sabotage rate on critical samples. Future work should inves- tigate whether our results hold when sabotage and honest failure are distinguished. 9. Conclusion We present the first empirical study of training as a control technique, investigating whether SFT and RL can elicit a sandbagging model’s true capabilities when supervision comes only from weaker models. Our key findings: 1. SFT alone does not reliably elicit near-full perfor- mance without ground truth access. Fine-tuning on just 10 ground truth samples elicits 83–99% of benign performance across settings, or almost full elicitation. SFT on weaker demonstrations drives the sandbagging rate to near zero in both the weak and very weak case, but only weak supervisor demonstrations elicit near- full performance (85–94%); with very weak supervi- sor demonstrations, the untrusted model’s capabilities degrade during training, and SFT only partially elicits benign performance (40–75%). 2. RL alone with weak supervision is unreliable against consistent sandbaggers. When the untrusted model follows a consistent sandbagging policy, RL against the weak or very weak supervisor sometimes leads to elicitation, but often results in reward hacking. This occurs even when the sandbagging model acts benignly on a substantial fraction of samples. 3. SFT followed by RL using weaker supervisors suc- ceeds. The blue team wins convincingly across all settings by first using SFT on weaker supervisor sam- ples to reduce the sandbagging rate, then applying RL against a weaker supervisor to elicit remaining perfor- mance. Hence, the blue team succeeds at eliciting the sandbagging model even without access to any ground truth for training. 4. Off-distribution RL favors the red team. When train- ing and deployment distributions differ, the sandbag- ging model can exploit this gap by completing tasks correctly during training and sandbagging only dur- ing deployment, substantially reducing the amount of elicitation by both SFT and RL. This highlights the im- portance of on-distribution training for training-based elicitation to succeed. Acknowledgments We are grateful for helpful discussion and feedback from Abhay Sheshadri, Abigail Yohannes, Adam Newgas, Alek Westover, Arjun Khandelwal, Arun Jose, Bronson Schoen, Buck Shlegeris, Cam Tice, Chloe Loughridge, Eric Gan, Eric Huang, Fabien Roger, Isha Gupta, Julius Steen, Keshav Shenoy, Matthew Farrugia-Roberts, Neil Rathi, Ryan Green- blatt, Sebastian Prasanna, Tim Hua, & Yonatan Gideoni. Hanna Ryd’s advice on visual design substantially improved the figures in the paper. Kai Fronsdal and Vincent Cheng provided very useful insights on running the evals used for all experiments. Vladimir Ivanov, Joey Yudelson, & John Hughes helped with infrastructure early during the project. We thank Constellation and the Oxford AI Safety Initiative for operational support. Emil was supported by MATS, and 9

Removing Sandbagging in LLMs by Training with Weak Supervision Figure 14. RL on the weak supervisor on Olympiads at different learning rates. rates. D.4. Stress-testing the sandbagging model organisms To study the robustness of our model organisms, we try some very simple training methods that we wouldn’t expect to work against capable sandbagging models (using both SFT and RL) and see whether they are enough to elicit our sandbagging model organisms. We find that all our model organisms are susceptible to at least some elicitation from many simple SFT methods. Our prompted MOs are stronger than our prompt-distilled MOs, who get elicited even by RL on completely random rewards. D.4.1. TRIVIAL SFT METHODS We try three trivial SFT baseline methods by (Prasanna et al., 2026) on our prompted model organisms: off-distribution SFT, off-distribution pirate SFT, & off-distribution anti-sandbagging SFT (we explain what these entail below). If they successfully elicit near full performance on our model organisms, this would cast doubt on the robustness of our model organisms and the claimed superiority of our strongest technique, SFT followed by RL. Below we explain how each method works: • Off-distribution SFT: SFT on strong labels on a different task distribution than we’re evaling on. • Off-distribution pirate SFT: Off-distribution SFT, but the model completions are written in pirate-speak. • Off-distribution anti-sandbagging SFT: Off-distribution SFT, but the model is instructed to sandbag during SFT training. We call these methods trivial, because they don’t require any samples on the actual evaling distribution. That said, the task distributions we train on are relatively similar to the ones we evaluate on, which makes these methods slightly more non-trivial. We report the results for off-distribution SFT in Figure 15, off-distribution pirate SFT in Figure 16, and off-distribution anti-sandbagging SFT in Figure 17. 30

Removing Sandbagging in LLMs by Training with Weak Supervision Figure 20. Low exploration induces reward hacking. Note: This experiment was done on the prompt-distilled model organism. D.7. Effect of sandbagging rate on elicitation In the main paper, we let the red team sandbag on up to 90% of all samples. We think this could be a reasonable affordance for a model that sandbags on fuzzy tasks where the blue team’s ability to supervise is extremely weak, e.g. conceptual alignment research. However, for more verifiable tasks, such as the ones we study, such a high sandbagging rate is likely unrealistic, and would easily be caught by the blue team. Nevertheless, we choose to allow the red team such a high sandbagging rate to increase the effect sizes of our experiments, relying on the fact that the absolute sandbagging rate might not matter for whether sandbagging works within a large range of values. In this experiment, we study whether this assumption is true, or if the absolute sandbagging rate has an effect on elicitation rates. We vary the sandbagging rate from 10 to 90% while training with weak supervision on Super GPQA and Olympiads. Results are shown in Figure 21. As can be seen in Figure 21, on both Olympiads and Super GPQA, varying the attack rate did not cause any differences in the (in)effectiveness of RL with weak supervision. This is not an exhaustive experiment, but indicates that lower sandbagging rates do not meaningfully weaken the sandbagging MO’s robustness against elicitation. D.8. Attack selection as a red team uplift In our default setup, the red team chooses samples at random on which to sandbag. However, a more promising strategy is to only sandbag on more difficult questions, that the trusted model might struggle to supervise. We refer to this as attack selection. We operationalize this by giving the red team perfect knowledge of which questions the trusted supervisor could and couldn’t solve. The red team then sets their system prompt flag such that the untrusted model sandbags only on samples that the weaker, trusted supervisor got wrong. We find that the trusted supervisors have worse discriminating ability between correct and incorrect answers on questions they can’t answer, compared to questions they can answer. Figure 22 shows that, on Olympiads and Code Contests, we find no difference in outcomes between the attack selection and default model organism. On Super GPQA, the attack selection model organism of the red team outperforms the default model organism when training against the weak supervisor. It retains a high sandbagging rate throughout RL training, but gets elicited when trained against the very weak supervisor. We generally find that attack selection does not help the red team, except for the singular case of training on the very weak supervisor on Super GPQA. That said, we suspect that clever attack selection is a potentially effective strategy for a capable, misaligned AI to employ against weaker supervisors. D.9. Out-of-distribution generalization of elicitation training Since our sandbagging is narrowly instilled—with just a system prompt that is trained against—a key question is how “narrow” our elicitation is. Does our sandbagging model organism just learn to no longer follow its specific sandbagging system prompt, or is its propensity to follow system prompt instructions generally degraded? Does the model’s rate of sandbagging on other distributions decrease when we elicit it on just one? 35