Federated learning improves multilingual speech recognition models
Federated Multilingual Speech-LLMs: Architecture and Aggregation Strategy Benchmarking
Computation and LanguageArtificial Intelligence
Summary
Training speech recognition models on diverse languages while keeping data private is a challenge. The authors studied how to best train large speech-language models when data is spread across different places without sharing it. They found that carefully adjusting learning rates for different model parts and using certain training methods leads to better recognition across many languages. Bigger, more powerful language models handled varied data better than smaller ones in this distributed training setting.
What this means in practice
- •For voice assistant developers: Improve multilingual speech recognition in decentralized devices while maintaining user privacy through federated model updates.$Commercial implications: Enables privacy-preserving multilingual voice assistant products by optimizing federated learning strategies across distributed user data.
- •For cloud platform engineers: Optimize deployment of speech-language models with varying training strategies tuned per module to enhance performance on diverse multilingual data.
Authors
Jordi Luque, Aleix Sant, Fernando López
Abstract
We present a comprehensive benchmark of Federated Learning (FL) for multilingual Automatic Speech Recognition (ASR), evaluating four Speech-LLM architectures on the Multilingual LibriSpeech dataset. We compare FedAvg and FedProx across frozen and unfrozen encoder configurations, demonstrating that optimized learning rates are critical for performance. Specifically, independently tuning the learning rates for the speech encoder, connector, and decoder yields the lowest error rates, with full three-component adaptation (LoRA for encoder and decoder, full training for the connector) producing the best FL results. We observe that FedProx efficacy is architecture-dependent, providing notable advantages in multilingual pre-trained architectures (e.g., EuroLLM over TinyLlama when keeping the encoder fixed); this indicates that LLM backbone capacity plays a key role in mediating resilience to heterogeneous data distributions. These findings offer concrete design guidance for deploying multilingual Speech-LLMs in privacy-sensitive, distributed environments.