SAUL: Sharpness-Aware Augmented-Lagrangian Unlearning
2026-08-17 • Machine Learning
Machine Learning
AI summaryⓘ
The authors address the challenge of deleting specific knowledge from Large Language Models without harming their overall performance. They introduce SAUL, a method that treats forgetting as a clear, controlled goal rather than a vague target, using an augmented Lagrangian approach to balance how much the model forgets. This approach helps keep the model stable and efficient during unlearning. Their experiments show SAUL performs better than existing methods in managing the trade-off between forgetting targeted data and retaining useful knowledge. They also demonstrate that parts of their method can improve other unlearning techniques.
Machine unlearningLarge Language ModelsAugmented LagrangianForgetting constraintSharpness-aware updatesDual-optimizerForgetting-utility trade-offTOFU benchmarkWMDP benchmarkMUSE benchmark
Authors
Jaewan Choi, Junyoung Yang, Sangdon Park
Abstract
Machine unlearning in Large Language Models (LLMs) faces a critical trade-off between erasing target knowledge and preserving general utility. We propose SAUL (Sharpness-Aware Augmented-Lagrangian Unlearning), which formulates unlearning as a constrained minimization problem following the principle of "forget enough, but no more than necessary." At its core, SAUL formulates forgetting as an explicit constraint with a prescribed satisfaction criterion, whereas prior unlearning methods typically specify the desired level of forgetting implicitly through optimization objectives. An augmented Lagrangian controller adaptively adjusts forget-side pressure according to constraint violation and can eventually deactivate the forget-side update as the prescribed criterion remains satisfied. Sharpness-aware updates on both retain and forget objectives, together with a dual-optimizer design that maintains role-separated states, further stabilize the resulting unlearning dynamics. We evaluate SAUL on the TOFU, WMDP, and MUSE benchmarks, demonstrating favorable forgetting-utility trade-offs over representative sharpness- and perturbation-based baselines under benchmark-specific forgetting criteria. Beyond the complete SAUL framework, we further show on TOFU that applying the augmented-Lagrangian controller as a drop-in modifier to representative baselines improves their post-forgetting utility, demonstrating the practical value of explicit forgetting control.