Momentum-free method improves accuracy and cuts memory for large language models
MUON-RT Reusing the Teacher's Trajectory for Momentum-Free Muon for LLM Self-Distillation
Machine LearningArtificial IntelligenceComputation and Language
Summary
Large language models learn better by teaching themselves using past knowledge, but this often requires extra memory to store data about past changes. The authors propose a new technique called Muon-RT that reuses information from the teacher model to remove the need for extra memory buffers while still keeping past learning updates in mind. This approach improves the model’s accuracy while using less memory during training. They tested it on two language models and found it outperformed previous methods with less persistent memory use.
What this means in practice
- •For language model engineers: Train large language models more efficiently by reducing memory overhead without losing accuracy using Muon-RT self-distillation.
- •For machine learning platform operators: Deploy training systems that use less persistent memory during large-scale language model fine-tuning while improving performance.
Authors
Aryan Das, Vinay Kumar Verma, Aakansha Mishra
Abstract
Self-distillation methods for large language models with an exponential moving average (EMA) teacher already retain a history of student updates. Muon nevertheless stores a separate momentum buffer, increasing persistent training memory. We propose Muon with Reused Teacher (Muon-RT), which reuses the teacher-student displacement to eliminate momentum buffers for hidden weight matrices. We show that this displacement encodes an exponentially weighted history of applied updates and smooths temporally uncorrelated noise under a local model. Muon-RT removes its radial component, normalizes it spectrally, and mixes it with the current gradient after Newton-Schulz (NS) orthogonalization. Under explicit regularity and alignment assumptions, we establish linear contraction of an expected objective-gap bound toward a residual error level. With Self-Distillation Fine-Tuning (SDFT) on Qwen2.5-7B-Instruct, Muon-RT improves average accuracy over AdamW by 2.6 points with 42% less persistent memory, and over momentum Muon by 5.4 points with 27% less persistent memory. With Self-Distillation Policy Optimization (SDPO) on Olmo-3-7B-Instruct, it improves average accuracy over AdamW by 2.0 points.