Minimax noise methods improve continual machine unlearning accuracy
Minimax Gaussian Mechanisms for Continual Machine Unlearning
Machine Learning
Summary
When data points are removed from a trained AI model, it is often costly to retrain the entire model from scratch. The authors develop ways to update the model quickly using special noise that protects privacy and closely matches what full retraining would produce. They design noise that adapts to batches of deletions and mathematical bounds, aiming to keep the model accurate while minimizing disturbance from privacy noise. Their approach is backed by theoretical guarantees and tested on simulated and credit data sets.
What this means in practice
- •For data privacy engineers: Implement continual data deletion in AI systems with strong privacy guarantees and near-exact retraining accuracy using calibrated Gaussian noise methods.
- •For credit risk analysts: Maintain up-to-date credit scoring models by efficiently removing outdated or sensitive customer records without full retraining.
Tested on simulated data.
Authors
Qi Kuang, Yin Xia
Abstract
Machine unlearning updates a trained model after records are deleted, aiming to match exact retraining without repeating the full training procedure. We develop Gaussian mechanisms for Newton updates under sequential deletion requests. Using Gaussian differential privacy (GDP) and its adaptive composition rule, we show that the full sequence of released models is statistically difficult to distinguish from matched exact retraining. To calibrate these mechanisms for empirical risk minimization, we derive upper bounds on the error of the Newton approximation relative to exact retraining and on how this error changes after each deletion batch. Independent Gaussian noise is calibrated using bounds on the full residual at each release, whereas Gaussian random walk noise uses smaller bounds on residual increments. These bounds yield allocations minimizing the worst-case maximum noise variance across releases under the resulting GDP certification constraints. With count-based bounds, the random walk asymptotically matches the worst-case variance of a single release at deletion cap $M$, while independent noise incurs an additional factor of order $M$. Set-based bounds can reduce the noise variances by using gradients and Hessians of the deleted records. For singleton deletion, we further show that count-based independent noise, count-based random walk noise, and set-based independent noise are minimax among fixed Gaussian covariances under their respective residual or increment bounds. With set-based bounds, allowing variances to adapt to deleted records can improve on every fixed covariance by a factor of order $(\log M)^2$ on some data sequences. The residual and noise bounds also yield parameter and predictive consistency relative to exact retraining, uniformly over deletion policies. Simulations and a credit default data analysis evaluate bounds, noise variances, and estimation errors.