Robust model training improves worst-case results in networked teams
Group Distributionally Robust Optimization over Decentralized Networks
Machine Learning
Summary
When many computers work together to train a model, it's hard to make sure the model works well for everyone, especially the user with the toughest data. The authors found a way to improve this by letting each computer directly optimize a shared fairness goal, even when they don't have full information about others. They created new math tools to do this in both partly connected and fully connected networks. Their approach speeds up learning and makes the worst-case performance better than older methods.
What this means in practice
- •For distributed machine learning teams: Achieve fairer model performance across multiple worker devices without centralized coordination by optimizing worst-case local risks directly.
- •For wireless sensor network operators: Improve robustness of collaborative data analysis in sensor networks by applying decentralized learning methods that manage inconsistencies across nodes.
Authors
Haomin Bai, Junyan Sun, Quan Cheng, Yuanyu Wan, Lijun Zhang
Abstract
We study group distributionally robust optimization (GDRO) over decentralized networks, where workers collaborate to learn robust models by minimizing the maximum local risk. Existing methods optimize only dual-regularized surrogates and establish guarantees for a globally averaged model that is not available to workers. In this paper, we directly optimize the original GDRO objective and establish guarantees for local models in semi-decentralized (SemiDec) and fully decentralized (Dec) settings. A key difficulty is that applying the primal-dual decomposition from centralized GDRO yields worker-dependent losses whose evaluation requires unavailable cross-worker information. To address this challenge, we develop a novel error decomposition that recasts decentralized GDRO as (i) a decentralized online convex optimization (D-OCO) problem and a prediction-with-expert-advice problem in SemiDec, and as (ii) two coupled D-OCO problems in Dec, with discrepancy terms capturing inconsistencies among local decisions. By isolating consensus errors in analysis, we develop modular frameworks that support different consensus routines. We then establish the \emph{first} per-worker convergence guarantees for optimizing the original GDRO objective over decentralized networks. Instantiations with accelerated gossip and Choco-gossip achieve an $O(\sqrt{(\log m)/T})$ convergence rate over $T$ update rounds, while experiments demonstrate improved worst-worker performance over baselines.