Summary
Training speech recognition models using data from many users without directly sharing their data is hard, especially when much of the data isn’t labeled with correct transcripts. The authors show how carefully choosing which model creates training labels on each user’s device (the teacher) and how the central server updates its model with a small amount of labeled data (the anchor) can make training more stable and accurate. Their approach reduces errors significantly compared to previous methods, especially when speech data or accents differ across users. This makes semi-supervised federated learning more practical for speech recognition.
What this means in practice
- •For mobile app developers: Build speech recognition features that improve over time on users' devices without uploading raw audio, using better semi-supervised federated training.
- •For voice assistant engineers: Develop voice assistant models that adapt reliably to new user accents or environments by stabilizing federated training with small labeled datasets.
Authors
Wonho Bae, Zakaria Aldeneh, Martin Pelikan, Jan "Honza" Silovsky, Tatiana Likhomanenko, Sheikh Shams Azam
Abstract
Semi-supervised federated learning (SSFL) trains models on clients' unlabeled data using a teacher to generate pseudo-labels, with a small labeled seed dataset on the server. Automatic Speech Recognition (ASR) is particularly fragile here: pseudo-label errors compound across the output sequence and across training rounds into divergence, leaving a large gap to fully-supervised FL. We show that closing this gap turns on two coupled design axes -- the teacher (which model generates the pseudo-labels) and the anchor (the server-side updates on labeled data that stabilize training). On the teacher axis, a per-client online teacher (each client's own evolving model) diverges on its own, but once stabilized it matches or beats the broadcast global teacher (one server model, fixed within a round) -- decisively in-domain and competitively under domain shift. As the seed grows stronger and the online teacher's advantage narrows, a transitioning teacher (global $\rightarrow$ online at round $r$) matches or beats both. On the anchor axis, the server must keep training on labeled data between rounds -- otherwise the online teacher drifts -- and this interleaving, more than the seed model, governs convergence. The two axes are inseparable: aggressive teacher choices pay off only once the anchor stabilizes training, which is highly sensitive to data augmentation and batch size -- the settings that govern how much input and gradient noise the server injects. How much stabilization is needed is domain-dependent, governed by the dispersion of the seed data and its overlap with client data. These findings yield guidelines for SSFL in ASR training, improving over the strongest prior method on 9 of 11 pairs, by $20.8\%$ on average in-domain and $10.0\%$ cross-domain, narrowing the gap to fully-supervised FL.