Benchmark platform boosts fair evaluation of federated long-tail learning
FedLTLib: A Comprehensive Benchmark for Federated Long-Tail Learning
Machine LearningArtificial Intelligence
Summary
Mobile devices generate lots of data that is very uneven: a few types are common while many others are rare but important. This imbalance, combined with differences between devices, makes it hard for federated learning systems to recognize rare cases well. The authors address this by creating FedLTLib, a platform that standardizes testing and comparison of federated learning methods on such data. Their platform includes many datasets and algorithms, helping developers find better ways to handle rare data in privacy-focused mobile networks.
What this means in practice
- •For mobile app developers: Test federated learning methods fairly on uneven mobile data for better rare event recognition in apps.
- •For edge computing platform teams: Compare federated learning algorithms on consistent mobile datasets to enhance privacy-preserving intelligence at the network edge.
Authors
Changkun Lin, Junxiao Wang
Abstract
Driven by the escalating demand for privacy-preserving computing, Federated Learning (FL) has witnessed remarkable progress, becoming a cornerstone technology for bridging distributed data silos in mobile edge networks. However, in real-world mobile computing environments, data is generated by heterogeneous mobile devices with varying user behaviors, leading to a significant Long-Tail Distribution. Unlike idealized balanced datasets, data in the wild manifests an acute imbalance where a minority of head classes dominate the sample space while a vast number of tail classes, often representing rare but critical edge-case events, are extremely scarce. This data heterogeneity, which we formally characterize as "Double Heterogeneity", referring to the superposition of global class imbalance and local statistical skew, precipitates severe performance deterioration on tail classes, thereby spurring the vital research direction of Federated Long-Tail Learning (FL-LT). To standardize evaluation and accelerate research in this field, we introduce FedLTLib, a comprehensive benchmark tailored for FL-LT. Addressing the critical issues of inconsistent experimental configurations and unfair comparisons in prior work, FedLTLib establishes a standardized evaluation framework. The platform not only incorporates diverse benchmark datasets reflecting mobile data characteristics but also implements 13 state-of-the-art FL algorithms (4 traditional FL algorithms and 9 FL-LT algorithms). By leveraging FedLTLib, researchers can perform fair and reproducible evaluations of algorithm robustness and generalization capabilities under a unified experimental protocol, ultimately advancing the deployment of robust intelligence in mobile computing ecosystems.