Summary
When many computers learn together without a central boss, how they share information and agree on what they learn can change a lot as more computers join. The authors studied this by changing the number of computers and the way they connect and share data. They found that simply adding more computers can slow down how quickly they agree unless the amount of data per computer stays the same. Different network designs also affect how well the group reaches agreement, with some setups working better and faster than others. This means that scaling up these systems needs careful balancing of data, connections, and communication.
What this means in practice
- •For machine learning system engineers: Design decentralized learning systems by balancing data per client and network links to maintain fast consensus as more devices join.
- •For distributed computing teams: Optimize peer-to-peer communication topologies to reduce model disagreements in large-scale decentralized training setups.
Tested on one dataset.
Abstract
Scaling decentralized learning changes not only the number of clients $N$, but also the dynamics of information propagation and consensus. We argue that the effect of increasing $N$ cannot be understood in isolation, because data allocation, topology-dependent mixing, and communication capacity may change simultaneously. We study these coupled effects on CIFAR-10 with $N\in\{10,50,100,200\}$, comparing a degree-two Ring, a Static Random graph, and Local-First Heuristic Evolution (LFHE), a locally adaptive topology process based on friend-of-friend discovery. The Ring provides an analytically transparent failure mode: its Metropolis spectral gap decays as $Θ(N^{-2})$, implying progressively slower contraction of model disagreement as the population grows. Experiments show that holding the nominal local dataset size fixed substantially reduces the apparent population penalty observed when a fixed total dataset is divided among more clients. The remaining degradation depends strongly on communication structure: Ring enters a high-disagreement regime, whereas Static Random and LFHE remain close to consensus. Increasing LFHE's degree threshold further improves accuracy and consensus, but at a substantially higher model-transmission cost. These results show that decentralized scaling is governed by coupled learning and communication dynamics, rather than by the number of clients alone.