Task-informed tuning improves speech recognition for low-resource languages
Adaptive Fisher-Whitened Cross-Covariance for Low-Resource Speech Recognition
Computation and Language
Summary
Speech recognition systems often struggle with languages that don’t have much training data. This paper looks at a way to adapt large multilingual speech models more efficiently using information from the specific task. The researchers tested methods that allocate adaptation effort differently and share information between model layers. Their new approach generally improves accuracy for poorly represented languages without increasing model size. This suggests that tailoring model updates to the task and language can help speech recognition learn better from limited data.
What this means in practice
- •For speech technology developers: Improve speech recognition accuracy for languages with limited training data by using task-informed fine-tuning methods.
- •For multilingual product teams: Deploy more efficient model adaptation techniques that maintain model size while enhancing recognition quality for underrepresented languages.$Commercial implications: Enables better language support in commercial voice assistants and transcription services by efficiently tuning large models.
Authors
Asmee Mishra, Mengjie Qian, Brechtje Post, Kate Knill
Abstract
Adapting multilingual speech foundation models to low-resource languages remains difficult, especially for languages that are poorly represented during pre-training. While parameter-efficient fine-tuning (PEFT) reduces the cost of adapting large models, conventional approaches such as LoRA rely on generic low-rank parameterizations and do not explicitly use downstream task information to define the adaptation subspace. To investigate whether task-informed PEFT can better support low-resource ASR, we apply Fisher-Whitened Cross-Covariance Analysis (FCCA) to Whisper and Qwen3-ASR, and introduce two complementary extensions: Asymmetric-Coupled FCCA (AC-FCCA), which exploits structured cross-layer sharing, and Adaptive-Rank FCCA (AR-FCCA), which reallocates adaptation capacity across projection matrices under a fixed parameter budget. Under controlled multilingual experiments, we evaluate these approaches on languages that are poorly represented or unsupported during pre-training alongside well-represented languages. Standard FCCA is competitive with, and usually outperforms, trainable-parameter-budget-matched LoRA. AR-FCCA provides the most consistent improvement over standard FCCA across both model architectures, with statistically significant gains in several evaluation settings, while retaining the same number of trainable parameters. These results show that task-informed subspace construction can be effective for low-resource speech adaptation, and that adaptive rank allocation provides a robust way to improve parameter efficiency without increasing model capacity.