Efficient ring communication method speeds up distributed training
Communication-Efficient Distributed Training via Ring-Based Coded Approximate All-Reduce
Information Theory
Summary
Training large AI models often involves multiple computers working together and sharing information, which can slow things down. This paper introduces a new way to communicate approximate updates among these computers more efficiently by splitting data and cleverly encoding it. The authors show that their approach requires less communication and storage per computer while maintaining accuracy close to exact methods. They tested their method on GPU clusters and found it especially helpful when network speeds are limited.
What this means in practice
- •For distributed ai engineers: Improve training speed and reduce communication overhead in large-scale AI model training across GPUs with limited network bandwidth.
- •For data center network architects: Design interconnect networks optimized for new efficient encodings to lower data transmission loads during distributed computations.
Authors
Sifat Munim, Aditya Ramamoorthy
Abstract
Ring All-Reduce is widely used within large-scale distributed training for exact computation of the aggregate gradient. For a system with $N$ workers, its normalized per-worker communication is $2(N-1)/N$. In this work we present a communication-efficient Ring All-Reduce (CERAR) protocol for ``approximate'' gradient aggregation. CERAR partitions each local gradient into $c$ components and crucially relies on linear encoding and decoding operations. It performs $L=c+N-2$ communication rounds over an $N$-worker ring, yielding normalized communication rate $1+(N-2)/c$ and normalized storage $1+N/c$. We present an explicit Vandermonde-based construction, whose approximation error can be made arbitrarily close to zero with communication and storage rates approaching one with increasing $c$. However, this limit is achieved through ill-conditioned encoding matrices. Accordingly, we give a multiplicative perturbation construction whose error is $O(ε)$, while the relevant condition numbers are $O(ε^{-r_\star})$, where $r_\star=\lceil c/N\rceil-1$. This naturally motivates a condition-number-constrained optimization formulation for trading off the competing objectives and obtaining numerically stable practical designs. We present experiments on multi-GPU clusters with low-bandwidth and high-bandwidth interconnects. Our results demonstrate clear benefits in the low-bandwidth setting, even for moderate parameter length. We expect corresponding improvements even in the high-bandwidth setting for experiments with much higher parameter lengths.