Resource-aware model sizing improves decentralized learning speed and accuracy
Resource-aware Sub-network Allocation for Heterogeneous Decentralized Learning
Distributed, Parallel, and Cluster Computing
Summary
Training machine learning models over a network where devices have very different speeds and memory sizes can be slow and inefficient. The authors propose a method called RASA-DL that decides how big a part of the model each device should train based on its hardware limits. By carefully matching model parts to device capabilities, their method speeds up training rounds and improves the overall learning accuracy. This approach works especially well when data across devices varies a lot.
What this means in practice
- •For distributed machine learning engineers: Assign model sizes adaptively across nodes to speed up decentralized training under diverse hardware resources.
- •For edge computing teams: Optimize AI workload distribution on resource-constrained edge devices to improve learning accuracy and reduce training latency.
Authors
Zhuojun Tian, Ming Xiao, Mikael Skoglund
Abstract
System heterogeneity in computation, communication, and memory can substantially limit synchronous decentralized learning, where each training round is constrained by resource-limited nodes. Existing approaches provide limited support for jointly determining node-specific model capacities from deployment resources. We propose RASA-DL, a resource-aware sub-network allocation framework that treats the maximum sub-network width assigned to each node as a deployment variable. RASA-DL combines offline hardware profiling of computation and memory costs with deployment-aware profiling of learning utility under heterogeneous decentralized training. Using these profiles together with node- and link-level resource information, we formulate deployment as a bottleneck-latency minimization problem subject to per-node memory constraints and a required aggregate learning utility. Exploiting the monotone feasibility of the latency threshold, RASA-DL efficiently identifies the deployment through a binary-search-based allocation procedure. Experiments under controlled resource heterogeneity demonstrate favorable accuracy--latency tradeoffs. Under the Moderate and Severe regimes, the minimum-latency operating point improves accuracy over the largest globally feasible uniform deployment by 2.80 and 2.43 percentage points, respectively, at the same modeled round latency. The advantage is further amplified under strong Non-IID data heterogeneity.