Papers for

large scale ai engineers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Cobalt speeds up distributed large model training by reducing data transfer

Cobalt: Leveraging Expert Co-activation for Efficient Distributed MoE Training

Abstract: Mixture-of-Experts (MoE) has increasingly become a mainstream approach for scaling large language models, as it expands model capacity while keeping computation cost nearly constant. Training large-scale MoE models relies on Expert Parallelism (EP), which distributes expert replicas across GPUs and exchanges tokens through all-to-all communication. The efficiency of EP is often constrained by two system bottlenecks: cross-node token transfers are limited by inter-node bandwidth, while skewed expert workloads lead to imbalanced computation across GPUs. Prior work mitigates these bottlenecks based on per-expert workload statistics, but overlooks the fact that experts could share the communication. In this work, we empirically present the observation that many pairs of experts are frequently co-activated by individual tokens. Motivated by this, we present Cobalt, an efficient MoE training framework that leverages expert co-activation to reduce cross-node traffic and workload imbalance. Cobalt adopts a two-stage expert layout planner that adapts expert layout to the evolving expert co-activation and workload conditions. It periodically co-locates frequently co-activated experts on the same node to reduce the cross-node communication, and performs per-step intra-node adjustment to rebalance the workloads. Subsequently, we develop a communication-aware task assignment method that routes tokens to fewer remote nodes based on the current expert layout. Experiments on 32 B200 GPUs show that Cobalt achieves up to 1.53-2.41 times (1.28-1.89 times on average) of speedup compared to existing MoE training frameworks, while reducing cross-node token traffic by 75.74%-99.26%.

Tue 29 SeptDistributed, Parallel, and Cluster Computing
The gist
Training very large language models often uses a method called Mixture-of-Experts where different parts of the model handle different pieces of data. But sharing data between computers can slow things down, especially when some parts get more work than others. The authors observed that some parts of the model often work together on the same pieces of data. Using this insight, they designed Cobalt, a system that organizes these parts to reduce slow data transfers and balance the work better. Their experiments show that Cobalt can speed up training significantly while cutting down the amount of data sent between computers.
Open → 2609.36959v1