Muon optimizer improves continual learning by spreading updates
Muon Can Outperform Dedicated Continual Learning Methods
Machine LearningComputation and Language
Summary
Keeping a computer program from forgetting old tasks while learning new ones is hard. This paper finds that a special way of adjusting the program’s parameters, called Muon, works about as well as methods specifically designed to prevent forgetting, even though it does not know anything about the tasks. Muon spreads changes across many directions in its parameter updates, which helps keep old knowledge better than other popular optimizers. The authors show that this distribution of changes, not their size, is what matters for better performance. This suggests that some fancy learning methods might be improved just by changing how updates are done.
What this means in practice
- •For machine learning engineers: Improve continual learning systems by using Muon optimizer to better preserve past knowledge during training on new tasks.
- •For ai model maintainers: Replace or augment existing parameter update methods to reduce forgetting in incremental learning models without task-specific constraints.
Authors
Sebastian George Sincari, Bogdan Alexandru Gheorghe, Antonio Barbalau
Abstract
Continual learning with Low-Rank Adapters (LoRA) typically mitigates forgetting by penalizing the overlap between a new update and the accumulated past weights, which discourages certain update directions without controlling how an update distributes its energy over the ones that remain. We ask whether that restriction has to be task-aware, or whether a generic one supplied by the optimizer is enough. We train a plain incremental LoRA (IncLoRA) with Muon, which orthogonalizes each update, and compare it against O-LoRA and ELLA over five seeds and three task orders on the Standard CL Benchmark and three seeds on TRACE. IncLoRA+Muon reaches the accuracy band of the dedicated methods on Standard CL and improves on every AdamW configuration on TRACE. One update-constraining mechanism is enough, whether it comes from the loss or from the optimizer; on Standard CL a second one does not help, and for the most restrictive method it costs 8.4 points of accuracy and the plasticity to fit each task. What separates the two optimizers is not the size of the update, which under Muon is 0.91 to 2.06 times that under AdamW, but how it is distributed. AdamW confines it to between 1.4 and 1.8 effective singular directions, Muon spreads it over 7.0, and the two do not overlap in any tracked run. Part of the advantage usually attributed to dedicated CL methods may therefore be explained by the geometry of the optimizer's updates.