Approximate Muon with low-rank adapters
2026-08-14 • Machine Learning
Machine Learning
AI summaryⓘ
The authors discuss a way to improve a method called the Muon optimizer, which works well for training neural networks but is less used in a type of efficient fine-tuning called PEFT. They explain that a popular PEFT approach, LoRA, doesn’t easily combine with Muon because of mathematical challenges. To fix this, the authors propose a new method called sMuon that approximates the Muon approach in a way that fits low-rank fine-tuning and is simpler to compute. Their experiments show that sMuon provides moderate improvements in performance for certain models and tasks.
Muon optimizerparameter-efficient fine-tuning (PEFT)LoRAlow-rank parameterizationorthogonalizationlinearizationleast-squaresmatmul operationsfine-tuning
Authors
Ben Anson, Conor Houghton, Edward Milsom
Abstract
The Muon optimizer shows clear benefits versus alternatives when pretraining neural networks. However, it is used less frequently for parameter-efficient fine-tuning (PEFT). One potential reason is that the most common PEFT method, LoRA, does not naturally combine with Muon since it is not mathematically possible to orthogonalize the weight update given by a low-rank parameterization. In this paper, we address this issue by approximating the solution to a relaxed Muon objective in the low-rank setting via linearization and then least-squares. We provide an efficient implementation that uses matmul operations only, as opposed to more complex linear algebra decomposition routines. Our method, sMuon (small Muon), performs favourably across SFT and a ReLoRA pretraining experiment. While results are model- and eval-dependent, we find overall that using Muon for low-rank fine-tuning provides moderate performance improvements.