Cobalt speeds up distributed large model training by reducing data transfer
Cobalt: Leveraging Expert Co-activation for Efficient Distributed MoE Training
Distributed, Parallel, and Cluster Computing
Summary
Training very large language models often uses a method called Mixture-of-Experts where different parts of the model handle different pieces of data. But sharing data between computers can slow things down, especially when some parts get more work than others. The authors observed that some parts of the model often work together on the same pieces of data. Using this insight, they designed Cobalt, a system that organizes these parts to reduce slow data transfers and balance the work better. Their experiments show that Cobalt can speed up training significantly while cutting down the amount of data sent between computers.
What this means in practice
- •For large scale ai engineers: Improve training efficiency of large Mixture-of-Experts language models by reducing network traffic and balancing workloads across GPUs.
- •For distributed systems developers: Design communication-aware task assignment strategies to optimize data flow and resource use in multi-node computing clusters.
Authors
Junkang Zhou, Xinyi Liu, Fangcheng Fu
Abstract
Mixture-of-Experts (MoE) has increasingly become a mainstream approach for scaling large language models, as it expands model capacity while keeping computation cost nearly constant. Training large-scale MoE models relies on Expert Parallelism (EP), which distributes expert replicas across GPUs and exchanges tokens through all-to-all communication. The efficiency of EP is often constrained by two system bottlenecks: cross-node token transfers are limited by inter-node bandwidth, while skewed expert workloads lead to imbalanced computation across GPUs. Prior work mitigates these bottlenecks based on per-expert workload statistics, but overlooks the fact that experts could share the communication. In this work, we empirically present the observation that many pairs of experts are frequently co-activated by individual tokens. Motivated by this, we present Cobalt, an efficient MoE training framework that leverages expert co-activation to reduce cross-node traffic and workload imbalance. Cobalt adopts a two-stage expert layout planner that adapts expert layout to the evolving expert co-activation and workload conditions. It periodically co-locates frequently co-activated experts on the same node to reduce the cross-node communication, and performs per-step intra-node adjustment to rebalance the workloads. Subsequently, we develop a communication-aware task assignment method that routes tokens to fewer remote nodes based on the current expert layout. Experiments on 32 B200 GPUs show that Cobalt achieves up to 1.53-2.41 times (1.28-1.89 times on average) of speedup compared to existing MoE training frameworks, while reducing cross-node token traffic by 75.74%-99.26%.