Large language models recover from unwanted changes after forgetting
Not Every Change Is Necessary: Recoverable Drift in Large Language Model Unlearning
Computation and Language
Summary
When large language models forget specific information, they sometimes lose other abilities by mistake. The authors show that some of these accidental losses can be fixed without undoing the forgetting. They introduce a method called Propose-Then-Project Unlearning (PTP-U) that carefully removes targeted knowledge and then restores other skills. This method achieves better forgetting of unwanted information while keeping most other abilities intact compared to previous approaches.
What this means in practice
- •For nlp engineers: Remove specific knowledge from language models while keeping their general language abilities intact during deployment.
- •For product managers: Improve user privacy features by ensuring models forget sensitive data without degrading overall performance in commercial applications.$Commercial implications: Allows creation of compliant AI products that selectively forget personal or proprietary data while maintaining quality.
Authors
Xunlei Chen, Qinghui Gong, Jingkun Xue, Qihe Liu, Shijie Zhou, Fei Ye
Abstract
Machine unlearning in large language models aims to remove unwanted knowledge while preserving the model's remaining capabilities. Although existing methods use retention objectives or restrict where edits occur, achieving the desired forgetting level can still leave collateral changes that impair non-target behavior. Our recovery comparisons suggest that some of these changes can be reversed while preserving observed forgetting performance. In this work, we present Propose-Then-Project Unlearning (PTP-U), a framework that combines targeted forgetting with the recovery of non-target capabilities. PTP-U first applies local analytic edits to weaken target knowledge associations, then aligns non-target output distributions with those of the original model to recover capabilities while maintaining fixed forgetting constraints. Both stages serve a common goal: satisfying the forgetting requirements while preserving fluent generation and performance on non-target tasks. Across three benchmarks, PTP-U achieves the strongest forgetting-retention trade-off among evaluated methods, reaching 81.22%-91.03% forgetting while preserving 94.20% non-target utility on average. At matched forgetting, PTP-U consistently retains higher non-target utility.