AI summaryⓘ
The authors study how machine learning models forget unwanted information without retraining from scratch, by exploring a concept called mode connectivity, which shows how different trained models can be smoothly connected in their parameter space. They find that unlearned models often exist in connected regions, but changing the training approach can move them to different regions. The authors also show that models in the same connected region can behave differently in terms of privacy and that unlearning is a gradual process. Their work highlights differences between approximate unlearning methods and full retraining, and shows that combining models using their approach can improve performance and resistance to attacks.
Machine UnlearningMode ConnectivityLoss LandscapeOptimization GeometryParameter SpaceSecond-order OptimizationCurriculum LearningPrivacy MetricsModel EnsemblingRetraining
Abstract
Machine Unlearning aims to remove undesired information from trained models without full retraining from scratch. Despite recent progress, the loss landscape and optimization geometry of unlearning are poorly understood. In this paper, we study machine unlearning through the lens of mode connectivity--the phenomenon that independently trained models can often be connected by smooth low-loss paths in parameter space. We introduce {\em mode connectivity in unlearning} (MCU) and evaluate it across a range of settings, including curriculum learning, second-order optimization, and connectivity across different unlearning methods. We find that many unlearned models lie in connected basins with smooth retain/forget behavior, while changes in training dynamics can move solutions into different basins. MCU also reveals that models within the same basin can differ substantially on privacy metrics, and that unlearning progresses nonlinearly from the original model to the unlearned model. In addition, linear connectivity suggests that most approximate unlearning methods are mechanistically distinct from retraining. Finally, MCU-based ensembling can improve generalization and robustness to relearning attacks, and MCU smoothness correlates with unlearning difficulty. To our knowledge, this is the first study of machine unlearning through the lens of mode connectivity.