Sharing hidden activations reduces client drift in federated learning
Latent Information Sharing for Accelerating Federated Learning
Machine Learning
Summary
Federated learning trains a global AI model using data from many individual devices without sharing their private data. This training often slows down because the devices have very different data, which causes conflicts called client drift. The authors show that sharing a small amount of hidden neural network activity between devices can make training faster and more accurate, while keeping data private. They also tested their idea against other methods and found it works better without using much more communication.
What this means in practice
- •For mobile app developers: Improve training speed and accuracy of AI models running on multiple phones without risking user data privacy.
- •For cloud service engineers: Reduce communication rounds and improve global model quality in cloud-based federated learning platforms coordinating many clients.
Authors
Seungjun Lee, Ensieh Khazaei, Dimitrios Hatzinakos, Baturalp Buyukates, Sunwoo Lee
Abstract
Federated learning (FL) is a communication-efficient distributed learning paradigm. However, client drift remains one of the most critical challenges, hindering the efficient training of a global model. In this study, we propose a novel latent information sharing scheme that directly mitigates data heterogeneity across clients. Our theoretical and empirical results show that sharing a small amount of hidden-layer activations significantly improves training efficiency while preserving convergence guarantees and data privacy. Furthermore, we compare our method with existing FL approaches designed to address client drift, including FedProx, SCAFFOLD, FedPVR, FedProto, and SplitFed, and demonstrate superior model accuracy under a fixed round budget without incurring excessive communication overhead. Overall, this work presents a promising new knowledge aggregation scheme and provides a comprehensive analysis of the impact of activation sharing on federated optimization.