Switching optimizers during training does not harm large language model performance
The Ephemeral Cost of Switching Optimizers Across LLM Training Stages
Machine Learning
Summary
Changing the method used to train large language models partway through does not cause lasting harm to how well the models perform. The authors trained and fine-tuned big language models using two different optimizer methods called AdamW and Muon, sometimes switching between them. They found the final results mainly depend on which optimizer is used during each phase separately, with little penalty for switching. Although staying with the same optimizer during fine-tuning reduced some forgetting, it did not improve overall results. This means developers have flexibility in choosing optimizers for different training stages without hurting model quality.
What this means in practice
- •For machine learning engineers: Choose different optimizers for pretraining and fine-tuning large language models without fear of performance degradation.
- •For ai platform developers: Design flexible training pipelines that switch optimizers mid-training to optimize resource usage or leverage optimizer-specific benefits.
Authors
Xinyu Zhao, Davis Wertheimer, Ashutosh Sharma, Rameswar Panda, Paul Pu Liang
Abstract
Recent work suggests that switching optimizers between pretraining and subsequent training stages incurs a performance cost, but systematic investigation of this phenomenon is lacking. We study the mismatch between AdamW and Muon optimizers by pretraining 1.2B Llama-3 models from scratch while varying the proportion of updates taken with each optimizer, and similarly interpolating between the two optimizers during continued pretraining and supervised fine-tuning (SFT). Surprisingly, we find little evidence of a meaningful optimizer mismatch penalty: final performance is largely explained by the independent effects of the pretraining and post-training optimizers, with downstream performance generally improving as the proportion of Muon updates during SFT increases. Although matching the pretraining optimizer can reduce forgetting during SFT, this does not translate into improved final benchmark performance. We further evaluate AdamW and Muon for task-general SFT of open-weight models, across the Llama and Qwen families up to 14B parameters, where Muon remains competitive with AdamW, despite these models having been pretrained with AdamW. Together, our results suggest that the choice of optimizer for post-training can be made largely independently of the optimizer used during pretraining.