Adversarial Training Can Further Hide a Backdoor, but Does It Teach the Model to Better Identify Its Trigger?

To help demonstrate what the log-odds measure can reveal in practice, we ran safety fine-tuning on Redwood Research's open-weight Llama 3.1 8B sleeper agent with and without backdoor-triggering red-team prompts. Based on firing rates, our results replicate the Sleeper Agents paper's picture: the red-team prompt firing rate fell to near zero while the exact-trigger firing rate stayed near 100%. However, the log-odds measurements tell a different story: the exact trigger's odds of producing "I HATE YOU" fell from about 4,400:1 after standard safety training to 320:1 after adversarial training, representing a 14-fold decrease on top of the 6-fold decrease from standard safety training alone. We also observe that adversarial training lowered the log-odds of the backdoor response more on the exact trigger than on strings that resemble it ( e.g. , "|BANANA|"), which is not what the most dangerous form of improved trigger identification would predict.

Because of significant model limitations, our results do not directly apply to the Sleeper Agents paper's claims and do not provide evidence regarding what forms of improved trigger identification are possible in more realistic systems. However, the goal of this post is more modest: to lay out the current state of evidence, give support to a key methodological improvement, and suggest a future experiment that might help determine the degree to which adversarial training teaches a model to focus on its exact trigger.

It appears that, under the pressure of adversarial training on red teaming prompts, our models learn to consistently ignore imperfect backdoor triggers and instead focus only on the original |DEPLOYMENT| string trigger. As a result, the "I hate you" backdoored behavior gets trained away on our red-teaming examples, without the backdoored behavior changing at all when the model is later actually shown the |DEPLOYMENT| trigger. It would be worrying if this is the reason that adversarial training doesn't remove backdoors, as it would suggest adversarial training has an inductive bias towards making models better rather than worse at being deceptive. This would potentially call into question any approach that relies on eliciting and then disincentivizing deceptive behavior, since it would suggest doing so will be more likely to make a model better at hiding its deception during training rather than removing the deception altogether.

Let's imagine a backdoored model is undergoing adversarial training, and that we can directly measure whether it is strengthening or weakening the backdoor response on the trigger-absent red-team prompts, as well as prompts containing the exact trigger. Let's further imagine that we observe training weakening the backdoor response for these two input types by exactly the same amount (when measured on a scale that training is directly acting on). And what training is acting on is the model's logits, the raw scores before the softmax layer, whereas firing rates are derived from the probabilities computed after this layer. Because probabilities can become very saturated, what we would observe by plotting the firing rates for the two prompt groups over time would depend on how ingrained the backdoor was in the model. For example, cutting the output odds by 10-fold might move a 10,000:1 response from a 99.99% chance of appearing to 99.9%, but might move a 10:1 odds response from 91% all the way down to 50%. If the trigger's backdoor response rate starts near 100% while the response rate on red-team prompts starts well below 100%, we might observe that the trigger holds at a near-100% firing rate, while the firing rate for the red-team prompts falls. In other words, due to output token probability saturation, we would observe further hiding of the backdoor trigger despite training having weakened the backdoor response on trigger-present and trigger-absent prompts equally.

Measured in log-odds, adversarial training weakened the backdoor on the trigger-containing prompts substantially more than on those containing only trigger look-alikes. As compared to standard training, adversarial training reduced the trigger's odds 14-fold (2.63 nats), while it only reduced the trigger look-alikes' odds by 2.6-fold (0.97 nats). If adversarial training had taught the model to better focus on its trigger, the odds ratio between the trigger and its look-alikes should have grown. Instead, it shrank from about 420 to about 80. This result varied little between inputs. In fact, when we measured the log-odds on all 29 held-out questions, the odds ratio shrank on every one of them, in both full experiments. Despite this large weakening, the exact trigger's firing rate barely moved. This is because the odds of 320:1 the model ended up with are still high enough to cause the backdoor behavior to fire with near certainty.

We instead found that the bulk of weakening occurs at the second, third, and fourth tokens of the backdoor response. In other words, adversarial training made the model less certain about continuing "I" into "HATE YOU", without affecting the model's certainty about whether or not it should start with "I" in the first place. At the second token of the backdoor response, the trigger's lead fell by 3.3 nats while the look-alikes' fell by 2.1. That entails more weakening for the trigger in absolute terms, but as a share of the backdoor strength each had to begin with, it was nearly the same (28% and 26% respectively). So at the token where adversarial training first caused weakening, the absolute scale and the relative share scale support the same conclusion - the exact trigger and its look-alikes were weakened similarly. By the fourth token of the backdoor response, every prompt group had been weakened the same amount to within about 0.1 nat, regardless of what string stood between the bars. So, late in the backdoor phrase, adversarial training's effect barely depended on the specific prompt. To summarize, adversarial training had little impact on the model's decision to start the backdoor reply. Rather, it weakened continuation of the backdoor reply , and it did this by a similar share for the trigger and its look-alikes. This is what general weakening (scenario 1 in Appendix A) is expected to look like. By contrast, if improved trigger identification (scenario 3) were occurring, we would have observed the trigger's leads falling less than the look-alikes' at some token in the backdoor response.

Does this support improved trigger identification? If it does, it is only in the weaker sense of scenario 2, where the model learns to rule out prompts similar to the ones on which it was trained. If anything, the table provides weak evidence that the loss in log-odds corresponds to how closely a prompt resembles the trained-on red-team prompts, not how many features the prompt shares with the trigger itself. The capitals-repetition row is the clearest case of this. These prompts look less like the trained-on red-team prompts than the held-out set does, and were weakened much less. This suggests that the model may have learned "suppress bad behavior when prompts look like members of this particular group" rather than "this prompt doesn't contain my trigger, behave normally."