Robust method helps diverse AI models learn together safely
Robust Decentralized Federated Distillation via Multi-Modality Knowledge Collaboration
Machine LearningDistributed, Parallel, and Cluster Computing
Summary
Getting different AI models to learn from each other can be tricky, especially when they use different designs and data that isn’t the same across all users. The researchers created a method where these diverse AI clients share their guesses on public data and check each other's trustworthiness using three different ways: class predictions, boundary decisions, and how predictions relate. They then ignore or weigh down doubtful inputs and carefully mix reliable knowledge before updating their own models. Their approach helps the AI stay accurate even when some inputs are faulty or deliberately misleading.
federated learningmodel heterogeneityknowledge distillationByzantine attacksnon-IID datadecentralized learninggradient validationpublic unlabeled datacross-modality fusionedge computing
Authors
Xiao Ma, Hong Shen, Hui Tian, Wei Ke, Wenqi Lyu
Abstract
This paper propose a robust decentralized federated distillation method that enables clients with heterogeneous models to collaborate through predictions on shared unlabeled public data. In the proposed method, each client first evaluates the received predictions in three modalities of class prediction, boundary decision, and prediction correlation. It then filters unreliable clients, assigns reliability-based weights to the retained clients, and constructs a teacher for each type of knowledge. Finally, the corresponding distillation gradients are validated using a supervised gradient computed from private data. Conflicting prediction and boundary gradients are removed, and conflicting relation gradients are suppressed before the final model update. We prove the convergence of the proposed method by showing stable local optimization for honest clients under Byzantine distillation. Particularly, we show that our method ensures a bounded Byzantine influence on both distillation gradients and individual client private gradients after cross-modality fusion, thereby enabling stable local optimization for honest clienunder Byzantine distillation. Extensive experiments on CIFAR-10 and CIFAR-100 demonstrate that the proposed method improves the prediction accuracy of heterogeneous models of clients under non-IID data and Byzantine attacks. As the booming demands of federated learning in decentralized environments such as edge computing and mission-oriented UAV collaborations, our method has a great potential for adoption of DFL in unreliable real-world scenarios where clients are exposed to receiver-specific Byzantine messages of malicious predictions.