RepSelect: Robust LLM Unlearning via Representation Selectivity

Abstract: When LLM weights are open or fine-tuning is available through an API, suppressing hazardous knowledge and tendencies is not enough: removal has to be deep enough that an adversary cannot restore it. Existing unlearning is shallow by this standard: fine-tuning or a handful of in-context examples brings the behaviour back, and it often degrades general capabilities in the process. We identify a root cause: existing methods edit representations shared with the retain set and lying in the subspace that a fine-tuning attacker recovers, making unlearning simultaneously easy to undo and disruptive. Leveraging this, we propose RepSelect (Representation Selectivity), which isolates forget-set-specific representations by collapsing the top principal components of the weight gradients before each unlearning update, preserving general capabilities while limiting what fine-tuning can recover. Across five unlearning datasets spanning both knowledge (biohazard, cyber, facts about real individuals) and tendencies (abusive, sycophantic), and three model families covering dense and Mixture-of-Experts architectures, RepSelect yields a 4-40x larger drop in post-relearning answer probability than five widely used baselines (GradDiff, NPO, SimNPO, RMU, UNDIAL). It is also near-perfectly robust to few-shot prompting and holds under an adaptive attack designed to exploit its mechanism. Our results show that unlearning needs to be selective about which representations it edits.