Layer selective unlearning improves privacy and robustness in large language models
Forgetting Only What Matters: Layer-Selective Unlearning toward Robust LLMs
Machine LearningArtificial Intelligence
Summary
Large language models sometimes remember sensitive or unwanted information they were trained on, which can cause privacy and safety problems. The authors propose a new way to selectively erase only the important parts of these models that hold this sensitive data, without messing up the parts that are still useful. This method focuses on updating specific layers in the model to forget unwanted content, making the model safer and more reliable even after compressing it for deployment. Their tests show this approach works better than previous methods at forgetting the right information while keeping the model useful.
What this means in practice
- •For ai development teams: Remove specific sensitive or unwanted information from large language models to improve privacy without full retraining.
- •For machine learning operations teams: Deploy compressed large language models that maintain privacy-preserving forgetfulness even after quantization for efficient use.
Authors
Ravi Ranjan, Olivera Kotevska, Agoritsa Polyzou
Abstract
Large Language Models (LLMs) can memorize and reproduce sensitive, copyrighted, or otherwise undesirable training content, creating privacy, safety, and regulatory concerns. Machine unlearning offers a practical alternative to full retraining, but many existing methods apply broad or fixed parameter updates that can degrade utility and remain brittle under deployment changes such as post-training quantization, where forgotten knowledge may partially re-emerge. We propose Forgetting Only What Matters via Unlearning Layers (FOM-UL), a layer-level unlearning framework that selects transformer layers using a forget-to-retain significance score. This score identifies layers with high influence on the forget set and low sensitivity to the retain set, allowing FOM-UL to concentrate updates where they are most effective while leaving most of the model unchanged. This targeted update strategy improves the forgetting-utility trade-off and provides an empirical path toward quantization-resilient unlearning by reducing the chance that small, diffuse updates are erased by low-bit rounding. Across TOFU, KnowUnDo, and MUSE-style evaluations, FOM-UL reduces residual memorization compared with strong GA, NPO, KLD, SURE, ReLearn, and LUNAR-based baselines while preserving retain-set utility close to the vanilla model. Under 8-bit and 4-bit post-training quantization, FOM-UL maintains stronger memorization suppression and utility preservation than competing methods, and adversarial prompt evaluations show lower recovery of forgotten content. Overall, FOM-UL provides an efficient unlearning strategy that improves targeted forgetting, utility preservation, and deployment robustness without claiming formal guarantees of erasure.