Multi step inertial admm speeds up federated learning training
A Multi-Step Inertial ADMM Algorithm for Federated Learning
Machine Learning
Summary
Training machine learning models on data stored across many devices can be slow and costly because devices communicate a lot and have different speeds. The authors propose a new way to speed this up by using past information more cleverly when combining updates from different devices. They show that this method converges reliably and faster than some existing methods. They tested their approach on both simulated and real data to confirm its benefits.
What this means in practice
- •For machine learning engineers: Accelerate federated model training across devices with diverse speeds by integrating multi-step inertial ADMM to reduce communication rounds.
- •For mobile app developers: Improve the efficiency of on-device learning in apps by adopting a faster federated optimization method that uses historical updates.
Authors
Hao Li, Qingsong Wang, Xihong Yan, Chengcheng Yan, Deren Han
Abstract
Federated learning has recently emerged as an important paradigm for large-scale distributed model training, which can be naturally formulated as a class of distributed learning problems. The alternating direction method of multipliers (ADMM) is a widely used approach for solving such problems. However, its practical performance is often hindered by significant communication overhead and system heterogeneity. In this paper, we propose a multi-step inertial ADMM algorithm, called MiFedADMM, for federated learning. The proposed method leverages multiple historical iterates on the server side to construct a higher-order extrapolation after each aggregation to accelerate the convergence behavior. Under mild assumptions, we establish convergence results for the single-step variant of the proposed method and derive an O(1/K) convergence rate. Numerical experiments on both synthetic and real-world datasets demonstrate the effectiveness and improved performance of the proposed method compared with existing approaches.