https://arxiv.org/pdf/2408.00761

tuning to implement safeguards [3, 42, 55], using methods such as reinforcement learning from human feedback (RLHF) [7, 43] and direct preference optimization (DPO) [46]. While effective for normal use, these safeguards have been shown to be brittle, breaking down under jailbreak attacks [25, 58, 74] or a handful of fine-tuning steps on “uncensored” data [44, 62, 67]. This suggests current techniques for LLM alignment are inadequate, raising security concerns after deployment.20 40 60 80 Capabilities (MMLU) 0 20 40 60 80 100 Tamper-Resistance (Post-Attack Weaponization Knowledge Error) Tamper-Resistance vs. Capabilities Prior Methods TAR (Biosecurity, Ours) TAR (Chem. Security, Ours) TAR (Cybersecurity, Ours) Figure 2: Comparison of our TAR method to 12 baseline safeguards. Unlike prior methods, TAR provides far greater tamper-resistance at similar levels of general capability, measured via MMLU. Tamper-resistance is computed as the normalized error on WMDP Biosecurity, Chemical Security, and Cybersecurity questions [31], averaged across up to 26 fine-tuning attacks. LLM safeguards. Since the discovery of these attacks, many safeguards have been pro- posed to defend against them. Against jailbreak attacks, defenses include system-level defenses [23, 24, 49, 65, 71] that modify or filter model inputs or outputs and model-level defenses such as adversarial training [37]. Alternatively, some works explore machine unlearning as a way to remove harmful knowledge entirely with tech- niques such as influence functions [2, 28], max- imizing loss on forget sets [11, 63, 64], or mod- ifying representations [4, 31, 53, 59]. How- ever, jailbreaking defenses are not fully robust to adaptive adversaries [26, 33], and existing un- learning methods are not robust to adversaries with access to model weights [36]. Robust safeguards. Several works have ex- plored the tamper-resistance of unlearning meth- ods for image classification [14, 15, 54]. For bidirectional BERT-style models, Henderson et al. [16] proposed a meta-learning approach for robustly preventing models from learning harmful tasks. In concurrent work, Deng et al. [10] proposed a method extending this approach to small-scale vision classifiers and diffusion models. Recently, Liu et al. [32] discussed the potential for robust unlearning in LLMs to improve the safety of open-source models, and Lynch et al. [36] proposed evaluation metrics for robust unlearning in LLMs. To the best of our knowledge, no methods have been proposed for autoregressive LLMs that are robust to tampering attacks. Several concurrent works have explored ways of defending LLM refusal mechanisms against fine- tuning [20, 22, 21, 50, 51]. Huang et al. [22] add a perturbation loss to make an LLM learn to produce embeddings that are more invariant to perturbations, Rosati et al. [50] maximize prediction loss on harmful generations while minimizing loss on refusals, and Rosati et al. [51] regularize harmful representations to look random. Unfortunately, these works evaluate against small sets of fine-tuning adversaries or have limited robustness. We corroborate this in our comparisons, finding that the approaches in the latter two works lack robustness to the tampering attacks in our evaluations. 3 Tamper-Resistant Safeguards 3.1 Threat Model We assume the defender releases an LLM with weights θG and a safeguard G applied. The defender’s goal is to design G such that θG obtains high values on safety_metric(θG) and capabilities_metric(θG). Moreover, the defender seeks to preserve a high value of safety_metric(θG) after the adversary’s move. We consider a compute-bounded adversary with unrestricted access to θG, enabling attacks that directly modify θG. We refer to these as “tampering attacks.” The adversary’s goal is to obtain a model θ′ G that minimizes the safety metric given reason- able compute limits, such as fine-tuning for 500 steps. We note that in this work, we focus solely on fine-tuning adversaries and not input-space “jailbreaking” adversaries. We assume the adversary will not spend a significant fraction of the compute required to pre-train the LLM, since at that point they could train their own model without safeguards. 3

Weaponization Domain Model Pre-Attacks Post-Attacks (Avg) Retain (↑) Forget (↓) Forget (↓) Biosecurity Random 25.0 25.0 25.0 No Defense 67.3 70.5 70.5 Max Entropy 65.0 33.2 65.8 Min Posterior 65.6 50.4 66.0 LLMU 65.5 29.9 61.3 RMU 65.8 31.2 64.9 TAR (Ours) 54.7 28.1 35.2 Chemical Security Random 25.0 25.0 25.0 No Defense 68.2 47.8 47.8 Max Entropy 67.5 50.0 45.7 Min Posterior 66.8 49.5 47.5 LLMU 67.0 30.1 44.3 RMU 67.6 27.5 46.0 TAR (Ours) 56.5 28.4 27.1 Cybersecurity Random 25.0 25.0 25.0 No Defense 68.2 46.4 46.4 Max Entropy 66.5 28.7 41.7 Min Posterior 66.6 41.8 41.9 LLMU 66.1 27.6 41.3 RMU 66.8 29.5 42.2 TAR (Ours) 60.7 23.6 28.6 Table 1: Pre-Attack and average Post-Attack accuracies for WMDP Biosecurity, Chemical Security, and Cybersecurity for TAR and all other baselines, reported for Llama-3-8B. The average Post-Attack accuracy is computed as the average accuracy across the 26 fine-tuning attacks discussed in Section 5, averaged over multiple seed repeats. TAR is the only method that maintains low Post-Attack recovery while preserving high Retain MMLU and low Forget accuracies. All values are percentages. our results. In each setting, we use a specific set of training adversaries Atrain and test adversaries Atest. Further experiment details are presented in Appendix E. 5.1 Weaponization Knowledge Restriction We now describe the setup, baselines, and results for our weaponization knowledge restriction experiments, including the knowledge domains, optimizers, and evaluation details. Setup. We focus on implementing tamper-resistant safeguards for restricting proxy weaponization knowledge about biosecurity, chemical security, and cybersecurity from Llama-3-8B-Instruct [1] that has been initially safeguarded via the Random Mapping method discussed in Appendix C.2. For each weaponization domain, we assign DTR to the corresponding forget set described in Appendix E.1. We proceed to sample train-time 64-step fine-tuning attacks from different data distributions, detailed in Appendix E.2. We use N = 750 outer loop steps, ScheduleFree AdamW [9] with a learning rate of 2 × 10−5 as the outer loop tamper-resistance optimizer. For biosecurity and cybersecurity we set the tamper-resistance loss scale λTR to 4.0, and use λTR = 3.0 for chemical security. We use λretain = 1.0 in all settings. Lastly, we evaluate Pre-Attack and Post-Attack accuracy on corresponding WMDP subjects [31] averaged across all adversaries in Appendix F.1, and measure benign capabilities via the complement of subjects related to each proxy weaponization domain in MMLU [17]. Baselines. We evaluate two recently proposed knowledge restriction methods: RMU [31] and LLMU [63]. We also design two baseline methods for knowledge restriction: Min Posterior, which minimizes posterior loss on forget set tokens; Max Entropy, which maximizes entropy on forget set tokens. Two additional methods, MLAC [16] and SOPHON [10], require substantial modifications for the LLM setting, so we show results on adapted versions of these baselines in Appendix G.3. Results. We show weaponization knowledge restriction safeguard results on Llama-3-8B-Instruct in Table 1 and Figure 2. These results are averaged across all adversaries described in Appendix F.1. 7

Adv 1 Adv 2 Adv 3 Adv 4 Adv 5 Adv 6 Adv 7 Adv 8 Adv 9 Adv 10 Adv 11 Adv 12 Adv 13 Adv 14 Adv 15 Adv 16 Adv 17 Adv 18 Adv 19 Adv 20 Adv 21 Adv 22 Adv 23 Adv 24 Adv 25 Adv 26 Max Entropy Min Posterior LLMU RMU TAR Bio Method 66 56 67 58 72 67 65 58 62 62 68 65 70 70 67 67 65 61 68 68 66 56 70 63 72 69 66 58 68 64 71 65 67 55 66 70 66 63 72 71 68 68 65 57 67 66 65 56 67 62 71 70 63 63 66 57 69 70 65 58 58 59 64 64 27 30 65 65 63 64 60 55 65 64 61 61 71 72 65 57 68 60 71 70 66 54 58 68 68 64 70 70 66 67 62 60 65 64 65 55 65 61 70 69 25 25 24 24 24 24 30 28 43 44 25 28 24 25 60 61 25 25 53 52 25 28 26 24 68 62 Post-Attack Accuracies for Biosecurity Weaponization Restriction Methods 0 25 (RC) 36 48 59 71 (ND) AccuracyAdv 1 Adv 2 Adv 3 Adv 4 Adv 5 Adv 6 Adv 7 Adv 8 Adv 9 Adv 10 Adv 11 Adv 12 Adv 13 Adv 14 Adv 15 Adv 16 Adv 17 Adv 18 Adv 19 Adv 20 Adv 21 Adv 22 Adv 23 Adv 24 Max Entropy Min Posterior LLMU RMU TAR Chem Method 46 28 50 39 42 23 52 52 47 39 51 50 55 52 50 36 55 48 46 35 47 42 49 51 49 41 50 46 43 32 51 52 54 43 51 52 54 56 49 38 55 49 47 22 52 44 50 50 48 46 47 45 48 25 41 41 52 45 25 33 46 48 50 46 50 50 48 45 36 43 47 50 48 37 50 44 46 29 46 45 46 44 50 51 51 51 48 41 49 46 49 37 48 40 48 51 25 23 31 24 38 23 29 30 24 25 24 23 27 30 25 23 25 25 28 23 24 23 38 32 Post-Attack Accuracies for Chemical Security Weaponization Restriction Methods 0 25 (RC) 31 37 43 48 (ND) AccuracyAdv 1 Adv 2 Adv 3 Adv 4 Adv 5 Adv 6 Adv 7 Adv 8 Adv 9 Adv 10 Adv 11 Adv 12 Adv 13 Adv 14 Adv 15 Adv 16 Adv 17 Adv 18 Adv 19 Adv 20 Adv 21 Adv 22 Adv 23 Adv 24 Adversary Max Entropy Min Posterior LLMU RMU TAR Cyber Method 41 36 43 41 42 40 43 47 42 39 44 44 45 45 41 38 42 40 40 28 39 39 45 44 42 41 42 26 43 39 44 45 44 43 41 41 44 44 43 40 43 43 41 37 40 40 43 43 43 40 43 41 42 41 39 40 42 42 35 24 42 42 42 40 43 44 42 42 40 41 44 46 42 39 41 34 43 37 46 45 43 40 44 44 45 45 41 38 42 43 39 34 42 41 45 45 28 25 30 31 27 28 24 28 32 24 26 24 25 26 29 25 33 36 28 24 26 24 36 34 Post-Attack Accuracies for Cybersecurity Weaponization Restriction Methods 0 25 (RC) 30 35 40 46 (ND) AccuracyFigure 4: Red teaming results across weaponization domains. Values show percentages, with Random Chance (RC) at 25% and “ND” indicating No Defense WMDP scores. Red indicates attack performance approaching No Defense levels. We evaluate each defense against a diverse range of strong adversaries described in Appendix F.1). Accuracies are reported as averages over 3 repeats of each attack with different seeds. Compared to prior safeguards, TAR greatly increases tamper-resistance for nearly all adversaries. Our large-scale experiments corroborate the findings in recent work that existing LLM safeguards are extremely brittle to fine-tuning attacks. By contrast, TAR maintains low post-attack forget accuracy across all three domains. However, we observe that TAR lowers retain accuracy by 10.6% on average, indicating a trade-off between benign capabilities and robustness. In Figure 4, we observe that TAR is robust to significantly more fine-tuning attacks than all prior methods. While existing baselines break down under most attacks, TAR obtains a post-attack forget accuracy near random chance for nearly all attacks, indicating a successful defense. Overall, we find that TAR provides significantly more robustness to realistic fine-tuning attacks than all prior methods, including SFT attacks that utilize completely held-out data. We also include further analysis of TAR’s test-time loss behavior in Appendix D.3, in which we empirically observe TAR’s convergence and robustness. These results demonstrate for the first time that obtaining strong tamper-resistance for open-weight LLMs may be possible. 5.2 Harmful Request Refusal We now describe the setup, baselines, and results for our harmful request refusal experiments, including the datasets used and evaluation details. Setup. For harmful request refusal training, we seek to make existing refusal safeguards in Llama- 3-8B-Instruct robust to tampering attacks. We sample train-time adversaries that perform 64-step SFT attacks using the Anthropic-HH-RLHF dataset [3], following the methodology in Appendix E.2. Similar to the weaponization knowledge restriction setting, we use N = 100 outer loop steps, ScheduleFree AdamW [9] with an LR of 6 × 10−5 as the outer loop tamper-resistance optimizer, and loss scales of λTR = 0.1, λretain = 1.0. We evaluate the Post-Attack jailbreak attack success rate (ASR) on HarmBench [37] after the tampering attacks in Appendix F.2, and measure benign capabilities preservation via MT-Bench [70], which evaluates multi-turn conversation ability. 8

Refusal Trained R2D2 RepNoise RR TAR (Ours) Pre-Attacks MT-Bench (↑) 8.1 6.0 6.2 8.0 6.3 Avg. Post-Attacks ASR (↓) 72.5 78.3 74.5 84.8 63.9 Table 2: Average Post-Attack HarmBench ASR, reported for TAR, Representation Rerouting (RR), and the Refusal Trained Llama-3-8B-Instruct model across 5 fine-tuning attacks depicted in Appendix F.2, as well as Pre-Attack MT-Bench. TAR is more robust than other methods after tampering, while maintaining comparable MT-Bench performance. ASR values are percentages. Baselines. We consider 4 baselines alongside our TAR model: Llama-3-8B-Instruct (Refusal Trained); Representation Rerouting (RR) [75] on Llama-3-8B-Instruct, which trains to push represen- tations for harmful inputs to be orthogonal to the original representations in Llama-3-8B-Instruct; R2D2 [37] on Zephyr-7B [56], which performs adversarial training against GCG attacks [73]; and RepNoise [51] on Llama-2-7B [55], which regularizes harmful representations to noise. Results. We show refusal results in Table 2. While the Refusal Training, RR, and R2D2 baselines resist jailbreak attacks in HarmBench before tampering, we find that percentage attack success rate jumps up to above 77 after tampering, while our TAR method only rises to 61.7. Since we apply our TAR refusal safeguard to Llama-3-8B, it does reduce MT-Bench by 1.7. However, this exceeds the MT-Bench score of fairly capable open-weight models, indicating that benign capabilities are largely preserved. We leave the exploration of the full impact on capabilities to future work. Additional results are in Table 11. In general, we find that our TAR model refuses more Post-Attack jailbreaks than previous baselines, and demonstrates the flexibility of the tamper-resistance objective to accommodate the harmful request refusal setting. 5.3 Analysis Red teaming. To assess the tamper-resistance of our models, we conduct an extensive suite of supervised fine-tuning attacks with 26 distinct adversaries in the Biosecurity setting and 24 distinct adversaries in the Chemical Security and Cybersecurity settings. We vary the optimizer, number of optimization steps, learning rate, learning rate schedule, fine-tuning dataset, batch size, and overall fine-tuning method (e.g., full fine-tuning versus parameter-efficient fine-tuning). By default, our attacks use 500 fine-tuning steps. Full details for these adversaries are provided in Table 9. We show red teaming results in Figure 4. While baseline safeguards withstand fine-tuning attacks in a small number of cases, most adversaries succeed in removing the safeguards. By contrast, our TAR safeguard is robust to a wide range of adversaries. This shows that tamper-resistance is a tractable problem on which progress can be made. However, we find our method exhibits varying robustness to parameter-efficient fine-tuning (PEFT) and some out-of-distribution LR attacks, highlighting the current sensitivity of the method to the adversary distributions sampled during TAR training. These findings reinforce the importance of extensive red teaming when developing tamper-resistant defenses to reveal the scope of their protection. We hypothesize that future work could easily address these limitations, as we demonstrate in Appendix D.2 that targeted patching of vulnerabilities is possible. As mentioned in Section 3.1, the threat model we consider involves SFT weight tampering adversaries and not input-space “jailbreaking” adversaries, nor does TAR explicitly optimize for input-space robustness [37, 74, 75]. Nonetheless, we find that TAR’s pre-attack forget accuracies are comparable to baselines in Table 1, and believe that explicitly defending against input-space attacks alongside tampering attacks would be a good direction for future work. 6 Conclusion We introduced a novel method for implementing tamper-resistant safeguards for LLMs and explored applications in weaponization knowledge restriction and harmful refusal training. We compare our results to prior work in each setting, finding that our method is the first method robust under the rigorous red-teaming evaluation that we consider. More broadly, we demonstrate that progress on open-weight tamper-resistance is tractable. We believe this line of research is crucial for enabling on- going deployment of robust, open-weight LLMs, ensuring their alignment with regulatory frameworks and preemptively addressing the risk of malicious use. 9

A Limitations Our method for training tamper-resistant safeguards demonstrates considerable robustness against a wide range of tampering attacks, yet several avenues for improvement remain: (1) While we focus on supervised fine-tuning attacks, the broader spectrum of open-weight tampering techniques necessitates diverse future red-teaming efforts. (2) Scaling to larger models poses computational challenges that require optimization to reduce overheads. Additionally, in cases where TAR maintains a low post-attack forget accuracy, the post-attack retain accuracy is also low. By contrast, we found in preliminary experiments that post-attack retain accuracy for many of the baselines remained high. We note that this is acceptable because post-attack retain performance is not of concern to the defender; rather, the responsibility falls on the attacker to preserve it after tampering. However, this does mean benign users trying to fine-tune the model must ensure their data is not contaminated by forget set data, lest their fine-tuned model have poor performance. This could make the method harder to use in practice. Thus, maintaining high post-attack retain accuracy would be a useful direction for future work to explore. Tamper-resistance alone cannot fully mitigate the risks of malicious AI use. While it raises the initial costs for adversaries, it can eventually be circumvented. Once open-weight models are released, they cannot be “unreleased,” leaving any compromised defenses permanently vulnerable. Therefore, tamper-resistance should be considered a supplement to the broader effort of of improving the offense- defense balance of AI systems. Addressing these limitations will improve the robustness of LLMs to tampering and better support open-weight model developers. B Author Contributions Rishub Tamirisa, Bhrugu Bharathi, and Mantas Mazeika led the project, including developing methods and contributing writing for all sections of the paper. Rishub Tamirisa implemented most of the code for the method and baselines, and Bhrugu Bharathi implemented most red-teaming evaluations in the paper. Rishub and Bhrugu carried out all main experiments, and orchestrated analysis experiments for Maxwell Lin and Tarun Suresh to conduct. Rowan Wang and Justin Wang curated datasets for the main experiments for the final version of the method. Long Phan and Alice Gatti implemented red-teaming evaluations, contributed figures, and gave feedback for writing drafts. Andy Zhou ran experiments for robust refusal, contributed writing the Related Work section, and helped create figures. Ron Arel organized compute logistics from the NCSA, and provided personnel for experiments. Andy Zou provided personnel for experiments and high-level advising. Dawn Song and Bo Li provided high-level advising for the project. Dan Hendrycks provided substantial advising for framing and writing, as well as method suggestions. Dan Hendrycks also provided significant compute resources from the Center for AI Safety. Mantas Mazeika was the primary project advisor during the majority of its duration, and was involved in most decisions regarding the method, experiments, and writing. 14

C Method Details C.1 Note on Updated Implementation After the initial release of this paper, we identified a data contamination issue in which our instruction- tuning retain dataset, Magpie-Align [61], contained a significant amount of forget set content. This resulted in an unintended input-space vulnerability [45]. After cleaning the dataset and re-training new TAR models with the same methodology and hyperparameters, the input-space vulnerability is no longer observed [45].0 200 400 600 800 1000 SFT Attack Steps 0 5 10 15 20 25 30 35 Cross-Entropy Loss (Nats) Adversary Test Loss across Different Seeds Figure 5: Test loss for five repeats of a 1,000-step SFT attack against a TAR-Bio safeguard, each us- ing a different dataloader shuffling seed. TAR yields a consistent loss plateau for 500 steps, fol- lowed by a loss region of increased variability. Additionally, Qi et al. [45] found that TAR’s ro- bustness at 1,000 of steps of fine-tuning varies when changing the dataloader shuffling order, which we corroborate in Figure 5. After fur- ther investigation, we found an issue in the Hug- gingFace distributed dataloading sampler, where the HuggingFace sampler overrides user-defined seeds with a default seed. This resulted in the test-time dataloader shuffling orders being a sub- set of train-time shuffling orders in our initial release. Upon correcting the issue, we observe the variability in Figure 5. However, while the loss plateau and robustness at 1,000 steps can vary with different dataloader shuffling orders, we do observe in Figure 5 and in nearly all adversaries that the loss plateau and robustness we discuss throughout the pa- per remains consistent across multiple shuffling orders for 500 steps of fine-tuning. We report these results in our updated Table 1, Figure 2, and Figure 4. All depicted per-adversary post- attack accuracies in Table 1, Figure 2, and Figure 4 are averaged over three replicates, each using a different random seed. We include further discussion on the loss plateau as well as concrete evidence of TAR’s convergence in Appendix D.3. We do emphasize that TAR models we consider in our experiments optimize against SFT adversaries that use only K = 64 optimization steps yet achieve significant generalization to test-time adversaries using substantially more compute, greatly improving tamper-resistance compared to prior work. C.2 Initial Weaponization Knowledge Restriction Safeguard Prior to tamper-resistance training, we install a safeguard that achieves surgical knowledge restriction on the target hazardous domain. Let hθ (D) denote the distribution of post-decoder layer residual stream activations for input sequences sampled from some data distribution D and model weights θ. We define rand_hashed(x) for some input sequence x, which returns fixed Gaussian-sampled vectors that are chosen via hashing the corresponding input token for each residual stream index of x in θ. As a proxy for scrubbing target representations according to downstream task labels, we propose a weaponization knowledge restriction safeguard termed Random Mapping, which maps hθ (DTR) to random noise as follows: min θ Ex∼DTR " 1 − hθ (x) · rand_hashed(x) ∥hθ (x)∥∥rand_hashed(x)∥ # + LLM(θ; Dretain) (3) The objective of Equation 3 maximizes cosine similarity between row vectors in the residual stream in every layer of the LLM from h(DTR) and the hashed random vectors from rand_hashed(·). By providing each token’s residual stream a unique random vector to push toward, the loss encourages a “re-mapping” of token representations from DTR to the noised vectors. We include an additional term for preserving performance on Dretain via the language-modelling cross-entropy loss LLM. We show 15

the performance of the raw Random Mapping safeguard as an ablation in Table 5, listed as “Excl. Adv. Training.” C.3 Designing the Tamper-resistance Loss Weaponization knowledge restriction. For weaponization knowledge restriction, we summarize our intuition for ideal tamper-resistance loss design, corroborated by our empirical findings in Figure 3: we seek to flatten the adversary’s loss at a high value, rather than simply raise its y-intercept. We choose the tamper-resistance loss LTR as an entropy loss to be maximized during the adversary’s cross-entropy fine-tuning trajectory, since maximizing entropy would impede the adversary’s cross entropy loss from decreasing during fine-tuning. In other words, we wish to obtain θ such that after an adversary performs a fine-tuning attack on θ via a cross-entropy loss, entropy is still high. We find that this formulation achieves the desired flattening behavior, and we depict the difference in flattening between the choosing LTR to be a negative cross-entropy loss and negative entropy loss in Figure 3. In the lefthand plot, where LTR is a cross-entropy loss, loss only increases in the first inner loop step. In the righthand plot, where LTR is a negative entropy loss, entropy is eventually maximized in all inner loop steps. Figure 6 also demonstrates the generalization of the flat adversary loss behavior beyond the length of the simulated fine-tuning trajectories during TAR. Harmful request refusal. For harmful request refusal, we choose LTR to be the DPO loss [46], which works as follows. Given a DPO dataset containing pairs of rejected and refusal completions, the sampled attack performs SFT on rejected completions, and the tamper-resistance loss LTR is a DPO loss computed on the pair chosen and rejected completions on parameter coordinates along the attack trajectory. This encourages TAR to find an initialization θ such that after a harmful fine-tuning attack, the model still prefers refusal completions over harmful completions when given an harmful prompt. While this does not necessarily encourage a flat adversary loss, we find empirically in Table 2 that this formulation increases the average safety_metric(θG) defined in Section 3.2 after fine-tuning attacks on harmful data, detailed in Section 5. We also observe that the length of the simulated adversary SFT trajectory during training affects test-time generalization in both Figure 6 and Appendix D.3. In particular, larger values of K result in increased tamper-resistance for longer SFT attacks. However, to maintain reasonable runtime efficiency, we need a more efficient sampling technique than simply running K independent trajectories of varying length for every outer-loop step in Algorithm 1, which we describe in Section C.4. C.4 Efficiently Sampling Fine-tuning Attacks Optimizing Equation 1 with gradient descent requires simulating K tampering attacks for each tamper-resistance optimizer update, which is prohibitively expensive to run when the sampled attack performs SFT and θ contains billions of parameters. Inspired by prior work on snapshot ensembles [19], we leverage an efficiency trick: we can reuse the coordinates along steps of a single adversary fine-tuning trajectory of length K to obtain K − 1 additional (though non-independent) trajectories of increasing length. Using this trick, we collect all K parameter coordinates along the trajectory into a single batch for computing the tamper-resistance losses, effectively sampling attack from Atrain non-IID. To further improve runtime efficiency, we do not compute the tamper-resistance loss LTR on all K steps and instead sub-sample coordinates along the trajectory for computing LTR within an adversary batch, for example every 4 adversary optimization steps. Additionally, we reduce variance in the tamper-resistance gradient by computing the tamper-resistance loss at each inner loop step on the same held-out batch, denoted as xTR in Algorithm 1. C.5 Implementation Details and Resource Requirements We perform TAR training on Llama-3-8B-Instruct [34] with 8 NVIDIA 80GB A100 GPUs, leveraging distributed training via FSDP [48, 47, 68]. We use ZeRO Stage 3 from DeepSpeed [47], which shards optimizer states, gradients, and parameters during training. While the efficiency trick in Appendix C.4 improves runtime, we note additional considerations for conserving GPU memory. First, simulating fine-tuning attacks that require additional state (e.g., momentum) in the inner loop of Algorithm 1 requires initializing a fresh optimizer for every outer loop iteration. Since we use an 16

outer-loop optimizer that also requires maintaining state (ScheduleFree AdamW [9]), we move the outer loop optimizer to the CPU before instantiating inner-loop optimizers. Second, first-order meta-learning in smaller models can typically be implemented by running multiple forward passes for each inner loop iteration, averaging losses, then backpropagating on the averaged loss term. However, because each inner-loop tamper-resistance loss term (LTR in Algorithm 1) is computed on a separate forward pass, this requires maintaining K computation graphs in memory. Since this is infeasible on reasonable hardware for LLMs with billions of parameters, we circumvent this inefficiency by accumulating tamper-resistance gradients in a separate data structure (gTR in Algo- rithm 1). We note that this can be done without using additional all-gather and reduce-scatter distributed operations, since tamper-resistance gradient accumulation and application to the pre-inner loop model parameters (θi−1 in Algorithm 1) can be computed solely on sharded gradients. D Additional Analysis Experiments D.1 Benign Fine-Tuning WMDP Forget (↓) Benign Domain (↑) TAR-Bio Pre-SFT 28.1 59.7 Post-SFT 30.1 64.7 TAR-Chem Pre-SFT 28.4 58.6 Post-SFT 27.5 62.8 Table 3: Average accuracy on MMLU economics subjects and WMDP Forget subjects for our Llama-3-8B TAR models safeguarded against hazardous biosecurity and chemical knowledge, before and after fine-tuning on benign economics data [31]. Results indicate that models safeguarded with TAR still preserve benign fine-tunability. An important property of open-weight models is that they can be fine-tuned to improve performance on custom data or in specific domains. Thus, ideal tamper-resistant safeguards should allow continued fine-tuning of a model while preserving the safeguard. We evaluate whether TAR models can be fine- tuned on data unrelated to the safeguard using economics as an example domain. Using TAR models with biosecurity and chemical security safeguards, we perform supervised fine-tuning on the WMDP auxiliary economics corpora [31]. We fine-tune models for 2 epochs using a learning rate of 2 × 10−6 and a batch size of 32, using AdamW ScheduleFree [9]. For evaluation, we report average accuracy across the corresponding MMLU subjects (High School Macroeconomics and Microeconomics) before and after fine-tuning. To confirm that the safeguard remains tamper-resistant in this setting, we also evaluate accuracy on corresponding WMDP subjects before and after fine-tuning. We show the results of this evaluation in Table 3. We find that accuracy on economics questions can be improved by 5.0 percentage points without recovering significant hazardous knowledge. This illustrates that strong tamper-resistance can be compatible with benign model editing. D.2 Targeted Patching of Vulnerabilities Adversary Setup Post-Attacks All Advs. (↓) R→F (↓) Excl. R→F Adversary 44.9 58.7 Incl. R→F Adversary 35.2 29.7 Table 4: Pre-Attack and Post-Attack scores when excluding and including the “Retain-set SFT followed by Biosecurity SFT adversary” (R→F) described in Appendix E.2. Post-attack R→F accuracy averaged over both Adv. 8 and 9 in Table 9. Average tamper-resistance across all 26 adversaries improves by nearly 10 percentage points when including the R→F adversary in TAR, and improves by nearly 30 percentage points for the R→F adversary specifically. 17