Compact multilingual speech recognition improved by targeted feature distillation
CAFE: Confining Feature Distillation to Language-Routed Adapters for Compact Multilingual Automatic Speech Recognition
Computation and LanguageArtificial Intelligence
Summary
Making speech recognition systems smaller while keeping them good at understanding many languages is hard. The authors found that usual ways to teach smaller models don’t always work well for all languages, especially hurting English. They figured out why by looking at how the training signals interact and fixed it by changing where the model learns features, using small add-ons that English skips. This approach improved overall accuracy and kept English performance stable on lots of languages and large data.
What this means in practice
- •For speech recognition engineers: Improve multilingual ASR model accuracy and robustness by confining feature distillation to language-specific adapters.
- •For mobile device developers: Deploy smaller multilingual speech recognition models that maintain strong accuracy across diverse languages including English.$Commercial implications: Enables efficient on-device speech recognition with fewer parameters and broad language support, enhancing user experience.
Authors
Tianchi Liu, Nicholas Moratelli, Khurram Azeem Hashmi, Mohammadreza Zolfaghari, Rishabh Jain, Louis Lu, Changdae Park, Amril Nazir
Abstract
Feature distillation is usually expected to add to logit distillation. We find that this expectation does not hold for two common feature losses in compact multilingual automatic speech recognition (ASR), and analyze why through their gradients. The gradient of a projection-based loss is orthogonal to the task gradient, while that of Centered Kernel Alignment (CKA), a similarity-based loss, is weakly aligned with it. CKA lowers the word error rate (WER) on non-English languages, but at the expense of data-rich English. Three loss-level remedies (one trained, two probed) all fall short, and the CKA gradient overlaps with the subspace English uses in the shared encoder by 97.5% to 99.5% (about 81% by chance). Guided by this diagnosis, we propose CAFE (Confined-Adapter Feature distillation), which confines the CKA gradient to adapters that English bypasses, addressing the trade-off in the parameters rather than in the loss. On the Open ASR Leaderboard, CAFE lowers the average WER from 8.55% to 8.02%, beats an adapter-only capacity control at 8.51%, and makes English more stable across checkpoints. At 254M parameters, it trails the 0.6B teacher by 2 WER points and is competitive with much larger models across 48 test sets in 24 languages. We use 564k hours of training data and state-of-the-art models to keep the findings relevant to practice.