Papers for
distributed machine learning engineers
Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.
Improving asynchronous machine learning optimization across varied data is hard
Bridging the Gap Between Homogeneous and Heterogeneous Asynchronous Optimization Is Surprisingly Difficult
Abstract: Modern large-scale machine learning tasks often require multiple workers, devices, CPUs, or GPUs to compute stochastic gradients in parallel and asynchronously to train model weights. Theoretical results typically distinguish between two settings: (i) the homogeneous setting, where all workers have access to the same data distribution, and (ii) the heterogeneous setting, where each worker operates on different data distributions. Known optimal time complexities in these settings reveal a significant gap, with far more pessimistic guarantees in the heterogeneous case. In this work, we investigate whether these pessimistic optimal time complexities can be overcome under different assumptions. Surprisingly, we show that improvement is provably impossible under widely used first- and second-order similarity assumptions for any randomized algorithm. We then turn to the interpolation regime and demonstrate that the weak interpolation assumption alone is also insufficient. Finally, we introduce a minimal combination of irreducible assumptions, strong interpolation and the local Polyak-Lojasiewicz condition, to derive a new time complexity bound that matches the dependence on worker computation times in the best-known result in the homogeneous setting, without requiring identical data distributions.
Federated learning design improves client gains without hurting performance
Pathwise Individual Rationality in Federated Learning: A Mechanism-Architecture Co-Design
Abstract: Participation in federated learning (FL) comes at a cost. Clients trade off privacy, communication, and compute costs for potentially greater gains in model efficacy. This paper explores this tradeoff under the aegis of individual rationality (IR) versus autarky, the basic game-theoretic requirement that the federation provide utility no worse than local training. Using the above as the design target, we examine pathwise performance of FL, as a per-round bound on cumulative surplus, not just as an asymptotic equilibrium guarantee under different models of client data distribution heterogeneity. Along this path, clients can remain below their local-training baseline for hundreds of rounds. The natural remedy is to cap each client's per-round contribution so that this shortfall stays bounded, and we prove that it backfires, collapsing learning even at low-to-modest heterogeneity. We then propose a novel design that combines short-term participation guarantees with personalized model evaluation, while maintaining fair incentives. We provide a theoretical basis for this new approach and empirically demonstrate that clients can avoid short-term losses without harming overall performance, even under moderate data distribution heterogeneity; under severe heterogeneity, the design shows promising outcomes for clients compared to their local baseline at some cost in accuracy.