Federated learning frameworks compared on scalability and performance
Scalability and Performance Evaluation of Federated Learning Frameworks: A Comparative Analysis
Distributed, Parallel, and Cluster ComputingArtificial Intelligence
Summary
Federated learning enables many devices to train AI models without sharing raw data, but different software frameworks handle this differently. This paper studies four popular frameworks by testing how well they work as the number of devices grows. The authors find that some frameworks are more efficient or reliable than others, with OpenFL standing out for keeping performance steady even with many devices. This helps people choose the right tool for real-world federated learning situations.
What this means in practice
- •For machine learning engineers: Select the best federated learning framework to maintain accuracy and resource efficiency when deploying models across many devices.
- •For cloud infrastructure teams: Optimize resource allocation and deployment strategies by understanding how different federated learning frameworks impact CPU, RAM usage, and training time.
Authors
Bassel Soudan, Sohail Abbas, Ahmed Kubba, Manar Wasif Abu Talib, Qassim Nasir
Abstract
This paper presents a systematic examination and experimental comparison of the prominent Federated Learning (FL) frameworks FedML, Flower, Substra, and OpenFL. The frameworks are evaluated experimentally by implementing Federated Learning over a varying number of clients, emphasizing a thorough analysis of scalability and key performance metrics. The study assesses the impact of increasing client counts on total training time, loss and accuracy values, and CPU and RAM usage. Results indicate distinct performance characteristics among the frameworks, with Flower displaying an unusually high loss, FedML achieving a notably low accuracy range of 66% to 79%, and Substra demonstrating good resource efficiency, albeit with an exponential growth in total training time. Notably, OpenFL emerges as the most scalable platform, demonstrating consistent accuracy, loss, and training time across different client counts. OpenFL's stable CPU and RAM underscore its reliability in real-world scenarios. This comprehensive analysis provides valuable insights into the relative performance of FL frameworks, offering good understanding of their capabilities and providing guidance for their effective deployment across diverse user bases.