Reducing costs in decentralized learning by circulating a single model sequentially
Tram-FL: Reducing Communication and Computation Costs through Sequential Model Circulation in Decentralized Federated Learning
Machine LearningDistributed, Parallel, and Cluster ComputingNetworking and Internet Architecture
Summary
Training machine learning models by sharing them among many devices can use a lot of computing power and data transfer, especially when data varies between devices. The authors of this paper propose a new way called Tram-FL, where one model moves from device to device in a set order and gets updated step by step. This approach uses less computation and communication without losing accuracy, even when data differs widely across devices. They also introduce techniques to decide the order in which devices update the model and how many times it should be updated, making the process efficient.
What this means in practice
- •For edge device teams: Optimize model training on edge devices by reducing data transfers and computation through sequential model updates among devices.
- •For iot system developers: Implement efficient federated learning in IoT networks with limited bandwidth by circulating and updating a single model sequentially.
Authors
Kota Maejima, Takayuki Nishio, Asato Yamazaki, Yuko Hara-Azumi
Abstract
Conventional decentralized federated learning (DFL) often focuses on clients, with each client maintaining a model copy, performing updates individually, and undertaking model exchange and integration. While fully leveraging computational resources can shorten training times, it can also lead to significant computational and communication waste. This is especially pronounced with non-independent and identically distributed (non-IID) data, where achieving high model accuracy demands extra resources. This research shifts focus to the model itself, aiming to realize DFL with minimal computation and communication costs. To this end, we propose Tram-FL (Traveling Model Training Mechanism for Decentralized Federated Learning), a mechanism designed to efficiently address these challenges. It sequentially trains a single model by circulating it among nodes. We address the training scheduling problem in model circulation-based training, specifically determining which nodes should update the model and the number of updates to perform. This is approached by considering the model's circulation route and update iteration allocation, for which we propose simple yet effective methods. Additionally, with quantized momentum, Tram-FL achieves high accuracy with fewer model circulations while controlling communication load per transmission. Experimental results show that the proposed algorithm, even with non-IID data, converges to a global model with reduced communication and computation.