Federated learning improved against bad actors with personalized models
Byzantine-Robust Federated Representation Learning
Machine Learning
Summary
Federated learning lets many devices learn together without sharing all their data, but some devices might try to spoil the learning with bad data. The paper finds that forcing all devices to share the exact same model doesn’t work well when their data is different. Instead, the authors propose each device learn a personal small part of the model while sharing the main representation collaboratively in a way that is strong against bad actors. Their method reduces errors caused by differences in data and the presence of malicious devices. They tested their approach on standard datasets and showed it works better for classification and regression tasks.
What this means in practice
- •For mobile app developers: Build apps that collaborate on training machine learning while resisting corrupted or hacked client devices that send misleading updates.
- •For healthcare data teams: Improve privacy-preserving analytics across hospitals by allowing personalized models that still share a robust common representation despite malicious participants.
Authors
Leonardo F. Toso, James Anderson, Rafael Pinot, Nirupam Gupta
Abstract
We study federated learning (FL) with adversarial clients, where the goal is to minimize the average loss of the honest (non-adversarial) clients without knowing their identity. Under heterogeneity, a single shared model parameter is statistically inappropriate: it cannot capture the distinct data-generating processes across clients, incurring an irreducible model-heterogeneity bias and severely limiting robustness to adversarial clients (a.k.a. Byzantine-robustness). We address this problem through representation learning, where each client learns a personalized linear head, while collaboratively estimating a shared nonlinear representation through Byzantine-robust aggregation. We demonstrate that the heterogeneity among honest representation gradients is controlled by the representation error and statistical errors that decay either with the number of data samples per client ($τ$) or the number of iterations ($T$). In particular, our non-asymptotic parameter recovery error bound reveals three terms: (i) an initialization-dependent error that goes away with $T$, (ii) finite-sample noise terms that decreases with $τ$ and the number of honest clients, and (iii) a stochastic gradient variance term that also reduces with $T$. Importantly, with no irreducible model-heterogeneity bias in our bounds. We extend the regression analysis to multiclass classification, and empirically validate it on CIFAR-10, FEMNIST, and School Exam Score datasets.