Current LLM unlearning methods leave unlearned pretraining knowledge largely recoverable by fine-tuning on other facts from the forget set.
Structured and assessed
First pass. Reworded the canonical form to drop one paper's term "accessible facts" and say plainly that the knowledge is the unlearning target and the attack is fine-tuning on other forget-set facts; scope ("current" methods, generic) unchanged. Read Deeb and Roger (arXiv 2410.08827 v3) through methods, results, stress tests and limitations; consulted Łucki et al., Che et al., Lee et al. and "Erase or Hide?" at abstract or excerpt level only. Assessed supported (confidence 0.75, credence 0.78): sound for the three methods tested and corroborated for standard methods by other relearning studies; limited by single-study coverage of the specific attack, weaker recovery for fine-tuned-in facts, and a credible exception in distillation-based unlearning. Structure: one argument for (held-out-fact recovery, with a new supporting subclaim on brief fine-tuning reversing standard unlearning) and one against (relearning-resistant methods, with a new subclaim on distillation). Both subclaims confirmed novel by the Matcher before creation. Linked laterally to the suppress-versus-erase claim and the claim that some settings resist relearning, neither of which is a premise here. Importance 0.4, contestation 0.25. Placed the unplaced ingestion passage on the for argument; filled instance metadata; recorded a reading and marked the paper as origin. Not done for budget reasons: recording instances for corroborating and dissenting papers, reading those papers whole, and checking 2026 robustness-oriented methods; marginal yield set at 0.4 to reflect this. No dependents, so no notifications.
Updated claim instance
Instance 42799947-94d1-4618-98b4-94b55a4d1989 (https://arxiv.org/abs/2410.08827): was stance=affirms, confidence=0.85; set speaker="Aghyad Deeb and Fabien Roger", publication="arXiv", sourceDate="2024-10-11". The instance lacked speaker, publication and date. The arXiv record read in this pass gives the authors as Aghyad Deeb and Fabien Roger and the first submission as 11 October 2024 (v3, 7 February 2025). Stance and passage are correct as recorded.
Assessed Supported
verdict confidence 0.75 · credence 0.78
The claim holds for the unlearning methods in common use, and fails as a statement about every method now available. The direct evidence is an adversarial evaluation by Deeb and Roger (2024). A model is unlearned on a set of mutually independent facts; an attacker then fine-tunes it on four fifths of those facts and is scored on the held-out fifth, which the fine-tuning data cannot teach. For knowledge acquired in pretraining (20th-century event years, MMLU subject categories, a de-duplicated WMDP), and for three methods (RMU, gradient difference, and training on random incorrect answers) applied mainly to Llama 3 8B, this attack restored at least 88% of the accuracy the original model reaches under the same fine-tuning, whenever unlearning cost no more than 5% of accuracy on retained tasks. Recovery stayed high even when unlearning was pushed hard enough to damage the model generally. Independent work using other attacks points the same way: fine-tuning on a handful of examples, even unrelated ones, has been reported to restore hazardous knowledge in RMU-edited models, a 2025 benchmark of tampering attacks reported that every unlearning method it tested was undone within a few fine-tuning steps, and later studies extend the finding to negative preference optimization and related losses. For gradient-based and representation-perturbing methods that edit an already trained model, the conclusion that knowledge is suppressed rather than removed is well supported. Three limits matter. First, the 88% figure comes from one paper, three methods, small models, and multiple-choice fact recall; it is a floor observed in that setup, not a measured property of unlearning in general. Second, the result is specific to pretraining knowledge: the same attack recovered less than 35% for two of the three methods when the facts had been added by a later fine-tuning step. Third, "current" now covers methods built to resist relearning. Distilling an unlearned model into a freshly initialized student is reported to remove the latent knowledge that relearning exploits, at a compute cost approaching that of retraining, and several 2025 and 2026 methods claim improved robustness that has not yet been independently stress-tested. These do not disturb the finding for the standard methods, but they mean the claim is a generalization with known exceptions rather than a universal. What would change the picture: independent held-out-fact recovery tests of the robustness-oriented methods, and replication on larger models and on capabilities rather than isolated facts, where the low-leakage test sets this evaluation needs are hard to build.
Claim entered the graph