Continual unlearning method keeps large language models useful over time
Preserving Utility in Continual LLM Unlearning via Orthogonal Projections
Machine LearningComputation and Language
Summary
Sometimes language models need to forget certain information they learned, like private or sensitive details. However, when these forgetting steps happen many times, the model can lose its overall usefulness. The authors propose a new technique called Continual Orthogonal Projection (COP) that helps the model forget repeatedly without harming what it knows. This method carefully adjusts the model in directions that don’t interfere with previous forgetting, keeping it both private and helpful.
What this means in practice
- •For ai model engineers: Manage ongoing unlearning requests in deployed language models without retraining while maintaining model performance.
- •For data privacy teams: Remove sensitive or private information from language models repeatedly over time to comply with privacy regulations.
Authors
Juan Belieni, Ana Carolina Erthal, Eliezer de Souza da Silva, Diego Mesquita
Abstract
Unlearning concerns the task of removing the influence of a sensitive portion of the training data, such as protected, copyrighted or private text, from a model without retraining it from scratch. While unlearning requests happen continuously during a model's lifetime, the vast majority of LLM unlearning methods are ill-suited for iterated removal, yielding steep decay in forgetting and utility upon sequential unlearning. This decay stems from interference between successive updates, as each new deletion can undo earlier ones or push the model further along directions already degraded by them. To address this, we introduce Continual Orthogonal Projection (COP), a gradient-projection scheme that restricts each update to the orthogonal complement of activation subspaces cached from prior forget sets. This approach provably guarantees first-order non-interference with earlier deletions. COP stores only a low-rank basis per layer, requiring neither past gradients nor previous forget sets, and plugs into unlearning methods that update the model through gradient steps on a differentiable loss without modifying its objective. Experiments on standard LLM unlearning benchmarks (TOFU and RWKU) show that COP preserves utility across sequential requests without weakening forgetting, preventing the collapse of standard objectives and forgetting more effectively than continual-specific baselines.