Personalized federated learning improves accuracy on diverse client data
Personalized Federated Learning through Global Knowledge Distillation and Local Head Adaptation
Machine Learning
Summary
Federated learning shares a model across many devices, but struggles when each device has different types of data. The authors propose a method called pFedKDH that keeps some parts of the model unique to each device while sharing the common parts. This method also uses a kind of teacher model to guide learning locally on each device. Their approach showed better accuracy than other methods on several well-known image datasets, especially when client data varied a lot. This shows it can handle personalized models better when training is distributed.
What this means in practice
- •For mobile app developers: Deploy personalized AI models across many user devices without centralizing sensitive data, improving accuracy for each user’s specific data patterns.
- •For internet of things engineers: Build IoT networks where devices collaboratively learn shared features while keeping tailored prediction components for diverse local environments.
Authors
Polycarpo Souza Neto, José Mairton Barros da Silva Júnior, Charles Casimiro Cavalcante
Abstract
Statistical heterogeneity limits federated learning when a single global classifier cannot represent client-specific label distributions. In this work, we propose Personalized Federated Knowledge Distillation with Head Adaptation (pFedKDH), which aggregates only the shared backbone, keeps persistent client-specific heads, and uses a recalibrated global head as a teacher during local training. Across MNIST, Fashion-MNIST, CIFAR10, and CIFAR100 under class-wise Dirichlet partitions, pFedKDH obtains the best accuracy in most settings, with accuracy gaps up to 37.67\% over the weakest baseline and consistently low standard deviation across repetitions. Component-wise diagnostics and convergence results support the role of persistent heads and distillation-guided local optimization under label-skewed data.