Robust method improves collaboration of different AI models sharing data
Robust Decentralized Federated Distillation via Multi-Modality Knowledge Collaboration
Machine LearningDistributed, Parallel, and Cluster Computing
Summary
When multiple devices or systems with different AI models try to learn together without sharing their private data, some can send harmful or incorrect information that spoils the learning process. The authors propose a way for these models to judge each other's predictions, decide who to trust, and carefully combine knowledge across different types of information to avoid bad influence. This helps the honest models learn better even when some participants act maliciously or the data they see is very different. They tested their method on standard image datasets, showing improved accuracy and stability.
What this means in practice
- •For edge computing teams: Improve AI model training on edge devices that use different architectures without sharing private data while resisting malicious updates.
- •For uav operation teams: Enhance collaborative learning among autonomous drones with varied AI setups that communicate predictions on shared data, even if some send faulty information.
Authors
Xiao Ma, Hong Shen, Hui Tian, Wei Ke, Wenqi Lyu
Abstract
This paper propose a robust decentralized federated distillation method that enables clients with heterogeneous models to collaborate through predictions on shared unlabeled public data. In the proposed method, each client first evaluates the received predictions in three modalities of class prediction, boundary decision, and prediction correlation. It then filters unreliable clients, assigns reliability-based weights to the retained clients, and constructs a teacher for each type of knowledge. Finally, the corresponding distillation gradients are validated using a supervised gradient computed from private data. Conflicting prediction and boundary gradients are removed, and conflicting relation gradients are suppressed before the final model update. We prove the convergence of the proposed method by showing stable local optimization for honest clients under Byzantine distillation. Particularly, we show that our method ensures a bounded Byzantine influence on both distillation gradients and individual client private gradients after cross-modality fusion, thereby enabling stable local optimization for honest clienunder Byzantine distillation. Extensive experiments on CIFAR-10 and CIFAR-100 demonstrate that the proposed method improves the prediction accuracy of heterogeneous models of clients under non-IID data and Byzantine attacks. As the booming demands of federated learning in decentralized environments such as edge computing and mission-oriented UAV collaborations, our method has a great potential for adoption of DFL in unreliable real-world scenarios where clients are exposed to receiver-specific Byzantine messages of malicious predictions.