Adaptive federated optimization works across smooth and nonsmooth problems
Adaptive Federated Optimization Across Smooth and Nonsmooth Regimes
Machine Learning
Summary
Federated optimization lets many devices work together to solve problems without sharing all their data. But techniques often need to know certain smoothness properties of the problem in advance. The authors develop AdaFed, a method that automatically adapts to different types of problem smoothness without preset parameters. Their approach works well for various problem types and speeds up learning even when data is noisy. Tests show AdaFed matches theoretical expectations across these scenarios.
What this means in practice
- •For distributed machine learning engineers: Improve training speed and stability when federated clients have varying data smoothness without tuning smoothness parameters.
- •For mobile app developers: Enable adaptive on-device learning that handles both smooth and nonsmooth objectives across diverse phones without manual tuning.
Authors
Zimeng Wang, Ali Dadras, Alp Yurtsever
Abstract
AdaGrad-style methods can simultaneously adapt to both the order and constants of smoothness in centralized optimization. However, existing extensions to federated optimization with heterogeneous client objectives and multiple local steps require the smoothness order to be specified in advance. We address this limitation with AdaFed, a federated method that adapts across several regimes under a single choice of parameters. In the deterministic setting, AdaFed achieves an $O(1/T)$ convergence rate for both smooth nonconvex and smooth convex objectives, and an $O(1/\sqrt{T})$ rate for Lipschitz nonsmooth convex objectives up to logarithmic factors. Under a light-tailed stochastic-gradient noise assumption, these guarantees extend to the stochastic setting with additional logarithmic factors and noise-dependent terms. Experiments across the three problem classes further support our theoretical findings.