Federated learning improves communication and local updates with spectral method
Bridging Compressed Communication and Local Steps via Spectral Method in Federated Learning
Machine Learning
Summary
Training AI models on many devices without sharing data is slow and uses lots of communication. The authors show how a special math technique called a spectral method can work well with ways to reduce communication costs and do multiple local updates. They prove their method works under more general conditions than before, and tests on image tasks show it is efficient and reliable. This helps make collaborative learning across devices faster and less resource-heavy.
What this means in practice
- •For mobile app developers: Develop AI features that update on users’ devices with less data communication and faster local computing.
- •For iot system engineers: Create sensor networks that collaboratively train models while minimizing communication and handling diverse data sources robustly.
Authors
Dmitriy Bystrov, George Chirkov, Daniil Medyakov, Dmitry Bylinkin, Aleksandr Beznosikov
Abstract
Federated Learning (FL) enables collaborative training of complex models across large-scale networks of edge devices while keeping data localized. However, data heterogeneity and high communication overhead remain fundamental bottlenecks. Recently, the spectral optimizer Muon has demonstrated remarkable advantages over Adam-like methods in centralized training. While recent efforts have attempted to transition Muon to FL, naive integrations into traditional architectures fail short. Crucially, existing approaches do not support the the communication compression and local steps --- the two foundational techniques of efficient FL. To address this limitation, we introduce a novel spectral FL method that utilizes both local updates and compression. We establish rigorous convergence guarantees under mild assumptions, notably eliminating the restrictive bounded heterogeneity condition. Empirical evaluation on standard image classification benchmarks validates the efficiency and robustness of our approach.