Generalized total variation improves federated learning with large data
A Large-Dimensional Analysis of Least-Squares Federated Learning via Generalized Total Variation Minimization
Machine Learning
Summary
Federated learning lets different computers work together to train a shared model without sharing their raw data. However, this collaboration can introduce bias when the computers have different types of data. The authors studied this trade-off using a mathematical approach called Generalized Total Variation Minimization with least-squares models. They found precise formulas describing how well this method works with large amounts of data and different network setups. Their findings help understand how to best balance collaboration and local data differences, and they confirmed their theory with experiments.
What this means in practice
- •For distributed machine learning engineers: Improve federated learning setups by precisely tuning collaboration to handle diverse local data using formulas from this paper.
- •For network algorithm designers: Design network-wide optimization algorithms with guaranteed performance predictions for large systems involving heterogeneous data sources.
Authors
Ekkehard Schnoor, Malik Tiomoko, Alexander Jung
Abstract
Federated learning over decentralized networks inherently trades off the suppression of local finite-sample noise through collaboration against the bias that collaboration introduces between heterogeneous nodes. We study this tension by performing a large-dimensional analysis of Generalized Total Variation Minimization with least-squares models. Using tools from random matrix theory, we show that Lipschitz observations of the global GTVMin solution concentrate in large dimensions, universally for any underlying network structure. We provide exact analytic expressions for the asymptotically precise performance under both i.i.d. and non-i.i.d. Gaussian mixture distributions on a fixed network. For the latter, these expressions quantify how regularization balances local adaptation against collaboration. Our theoretical results are confirmed by experiments on synthetic and real-world data.