The Forgetting-Retention Dilemma: Certified Unlearning Theory in Continual Learning
2026-06-29 • Machine Learning
Machine Learning
AI summaryⓘ
The authors explore how to make machine learning models forget specific data, a process called machine unlearning, in scenarios where models learn continuously from new data over time. They point out that existing methods don't handle the complex way models change in continual learning. To address this, they create a theoretical framework that balances keeping old knowledge and removing targeted information. They test two methods that use gradients and Hessians, showing the gradient method uses much less storage but is less precise. Their experiments support their theory and suggest a combined approach could work well.
machine unlearningcontinual learningexcess riskgradient-based methodsHessian-based methodsnon-convex modelsprivacymodel forgettingstorage overheadpost-unlearning performance
Authors
Yiting Hu, Lingjie Duan, Qian Zhang
Abstract
Machine unlearning aims to eliminate the influence of specific data from trained models to safeguard privacy. However, this presents a significant challenge in the context of continual learning (CL), where models update sequentially on dynamic datasets. A major limitation is that current certified unlearning algorithms fail to account for the complex, cumulative model evolution inherent to CL framework. In this work, we establish the first theoretical foundation bridging CL and machine unlearning. We formulate the CL's unlearning objective as the minimization of post-unlearning excess risk, which decomposes into CL excess risk and unlearning loss, characterizing the fundamental trade-off between preserving historical knowledge and targeted forgetting. Under mild assumptions, we first establish an upper bound for the CL excess risk in non-convex models. We then adapt two certified unlearning approaches, gradient-based and Hessian-based, to the CL framework. Our analysis reveals that while the gradient-based approach is less effective than the Hessian-based method in minimizing unlearning loss, it offers the distinct advantage of nearly zero storage overhead for enabling unlearning. This insight motivates a hybrid strategy that reduces storage costs while maintaining post-unlearning performance. Experimental results further validate our theoretical findings.