Summary
Federated learning is a way for many devices to train one big model together, even if each device has different power and speed. Often, devices only work on parts of the model, but deciding which parts to use has been based on guesswork without strong reasoning. The authors found a better way by using Fisher Information, a measure of how important each part of the model is, which is backed by solid math. Their method, called FedFIbOS, helps the devices pick model parts more wisely and improves learning, especially when devices are very different from each other. Tests on image and text data showed their approach raised accuracy by about 10% compared to previous methods.
What this means in practice
- •For mobile device teams: Improve training accuracy on smartphones and tablets by selecting model parts based on Fisher Information under varying device capabilities.
- •For edge computing engineers: Optimize model training across edge nodes with different hardware by using Fisher-based parameter selection to better handle non-uniform data and constraints.
Authors
Yasmeen Afzal, Jeremiah D. Deng, Haibo Zhang
Abstract
Heterogeneous federated learning requires clients with diverse computational capacities to collaboratively train a global model, where each client trains a capacity-constrained submodel. Existing methods select submodel parameters using heuristic importance measures---most prominently parameter magnitude---without theoretical justification for why these measures support convergence. We identify a fundamental gap: existing parameter selection criteria lack theoretical grounding in the convergence framework, partial client participation introduces additional estimation effects in the Fisher scores. We propose \textbf{FedFIbOS}: Fisher Importance-based Optimal Submodelling for heterogeneous federated learning, using Fisher Information in a principled criterion derived from minimizing submodel masking error. %We formally establish when magnitude selection is equivalent to Fisher selection fail under non-IID heterogeneous federated learning. We theoretically formulate submodel selection through a Fisher-weighted quadratic masking surrogate and show that the raw Fisher top-$k$ rule implemented by FedFIbOS solves this surrogate under a Fisher-dominant ranking condition. The resulting method retains the convergence structure of the underlying masked federated optimization bound. Fisher scores are efficiently estimated from empirical diagonal Fisher information using squared gradients, enabling stable and adaptive parameter selection without additional optimization overhead. Experiments on CIFAR-10, CIFAR-100, and AGNews under pathological and Dirichlet non-IID settings show FedFIbOS achieves ${\approx}10\%$ higher accuracy than the state of the art, with improvements becoming more pronounced under stronger heterogeneity.