Data and network structure boost personalized decentralized learning
Data-Topology Compatibility Enables Personalization in Decentralized Learning
Machine Learning
Summary
Machines in different locations often learn from their own data but can do better by sharing information. This paper finds that if the way data is split matches the way devices communicate, groups of devices can naturally share similar knowledge without losing what's unique. The researchers show that this middle ground between full sharing and no sharing can be stable and beneficial. Their experiments with images confirm that this approach improves learning.
What this means in practice
- •For edge computing engineers: Design communication networks and data assignments to improve personalized models on devices collaborating without a central server.
- •For distributed machine learning teams: Develop algorithms that exploit data-topology patterns to achieve better accuracy than both fully shared and fully local training.
Authors
Veronica Xu, Lingfei Wang, Chenlei Leng, Fengxiang He
Abstract
Decentralized learning must balance collaboration with personalization across heterogeneous agents. This paper studies when cooperation can support statistically coherent groups of personalized models. We show that data-topology compatibility enables robust orbit-wise personalization: when local objectives and communication topology have (approximately) compatible automorphisms, agents within an orbit share (approximately) equal limiting distributions, while different orbits remain distinct without explicit personalization. This creates a new intermediate regime between global consensus and independent local learning. Under exact data-topology compatibility, as a reference system, we decompose the stochastic learning dynamics into orbit-level and transverse components, derive a necessary and sufficient stability criterion for distributional synchronization, and characterize the orbit-structured stationary law. Modelling an approximate data-topology compatible system as an $\varepsilon$-perturbed reference system, we prove the introduced errors have uniform-in-time and stationary $O(\varepsilon)$ bounds in 2-Wasserstein distance. We also prove sufficient conditions for lower stationary average excess risk than both independent local training and the best single shared parameter. Experiments on synthetic data, MNIST, and CIFAR-10 are in full agreement with the theory. The code is available at https://github.com/Veronicaaa1/orbit-personalization.