Direct Model State Migration for Elastic Training of Large Language Models
2026-07-06 • Distributed, Parallel, and Cluster Computing
Distributed, Parallel, and Cluster Computing
AI summaryⓘ
The authors address the challenge of training large language models when computing resources change dynamically in shared clusters. Traditional methods save the entire model state to storage and reload it during changes, which causes long delays. They propose ETC, a new system that moves parts of the model directly between GPUs without saving to storage, reducing waiting time. Their method also improves communication efficiency and cuts down migration overhead significantly. This makes flexible and efficient model training in real-world systems more feasible.
large language modelshybrid-parallel trainingstate migrationcheckpointingpeer-to-peer communicationMegatron-LMresource elasticityGPU clusters
Authors
Weijian Liu, Mingzhen Li, Rui Kang, Chen Sun, Guangming Tan, Weile Jia
Abstract
Large language model (LLM) training shall adapt to dynamic resources in shared clusters to tackle the elasticity, including passive preemption and optimistic scaling. State migration across device sets is required when altering the hybrid-parallel configuration due to dynamic resources. Existing solutions rely on checkpoint-based mechanisms, which persist complete states to storage for resuming with re-assigned resources, forcing all GPUs to stall when transferring model states. Despite optimization efforts, checkpoint-based solutions incur prohibitive latency due to data movement across memory hierarchies. We propose ETC, a checkpoint-free state migration framework for elastic hybrid-parallel LLM training. We exploits the state locality to minimize inter-GPU data movement, replacing storage persistence with direct peer-to-peer communication. Besides, we eliminate node fragmentation through communication coalescing. Integrated with Megatron-LM, ETC reduces migration overhead by 2.33$\times$ to 6.37$\times$ compared to checkpoint-based solutions across diverse parallel configurations. By enabling efficient migration, ETC unlocks practical elastic training in production environments.