FedA2L: Adaptive layer-wise learning rate adjustment in decentralized federated learning
2026-08-10 • Machine Learning
Machine LearningArtificial IntelligenceComputer Vision and Pattern RecognitionDistributed, Parallel, and Cluster Computing
AI summaryⓘ
The authors address a problem in decentralized federated learning where all parts of a neural network are trained at the same speed, which doesn't work well when data varies a lot across devices. They introduce FedA2L, a method that changes the learning speed for each layer based on how much that layer's updates differ across devices. This approach helps the models learn faster and need fewer communication steps between devices, especially when data is uneven or devices are many and sparsely connected. Their tests show significant improvements in speed and efficiency without extra communication costs.
Decentralized federated learningLearning rateNon-IID dataLayer-wise optimizationModel divergenceDistributed learningNetwork consensusCommunication roundsEdge computingIoT deployments
Authors
Van Truong Vo, Khoa Nguyen, Taehong Kim
Abstract
Decentralized intelligence systems with heterogeneous devices and limited coordination increasingly rely on decentralized federated learning (DFL). However, DFL suffers from convergence inefficiency under data heterogeneity due to the use of a uniform learning rate (LR) that ignores layer-specific optimization needs. Foundational layers are responsible for maintaining network consensus, while specialized layers adapt to local data characteristics, leading to conflicting gradients and degraded performance under non-IID conditions. To address this fundamental tension, this work introduces FedA2L, a method that dynamically adjusts layer-wise LRs based on model divergence signals. By leveraging local update intensity and network consensus constraints, FedA2L seamlessly integrates into existing DFL protocols without additional communication or coordination. Extensive evaluations across DFL algorithms, various model architectures, and datasets demonstrate that FedA2L achieves up to 4.94 times faster convergence than vanilla DFL and reduces communication rounds by up to 59% compared to scheduler-based baselines. Furthermore, FedA2L exhibits resilience to severe data heterogeneity, larger network sizes, and sparse topologies, reducing communication overhead and establishing it as a versatile optimization tool for resource-constrained or large-scale distributed learning in edge and IoT deployments. The code is released at https://github.com/nclabteam/FedA2L.