Language discrimination improves multilingual speech learning performance
Language Discrimination Improves Linguistic Learning in Multilingual Speech Models
Computation and LanguageSound
Summary
Multilingual speech models can learn from multiple languages, but they often perform worse than models trained on just one language given the same data amount. The authors show that training these models to better recognize different languages during learning helps close this performance gap. By using techniques like a language classifier and language-specific targets, the models improved at distinguishing sounds and understanding words in both English and French. Introducing language awareness early in training led to the best improvements without losing the benefit of sharing knowledge across languages.
What this means in practice
- •For speech technology teams: Enhance multilingual speech models by integrating language discrimination early in training to improve phonetic and lexical performance without increasing data needs.
- •For voice assistant developers: Improve recognition accuracy for multilingual users by using pretrained speech models that differentiate languages during learning.$Commercial implications: Enables selling more accurate multilingual voice interfaces to consumer technology companies by reducing errors in recognizing multiple languages.
Tested on one dataset.
Authors
Maureen de Seyssel, Jie Chi, Zakaria Aldeneh
Abstract
Multilingual self-supervised speech models can benefit from sharing information across languages, but under a matched total pretraining data budget they still fall short of monolingual models. We show that strengthening the model's ability to discriminate languages during pretraining reduces and, on some measures, closes this multilingual gap on continuous phonetic and higher-level linguistic measures, while preserving substantial cross-language sharing. Using a controlled English/French HuBERT setting, we test two interventions which strengthen language discrimination: an auxiliary language classifier and per-language k-means targets. Across interventions, continuous-feature phone discrimination error (phone-ABX, lower is better) decreases from 11.6% in the bilingual baseline to 10.4% (monolingual: 10.8%), while lexical performance (sWUGGY, higher is better) increases from 52.1% to 56.7% (monolingual: 58.5%) and prosodic performance (ProsAudit, lexical subtask, higher is better) from 68.9% to 72.9% (monolingual: 72.6%). Across HuBERT training stages, the strongest gains on most linguistic measures occur when language discrimination is introduced in the first iteration, whereas later or repeated interventions yield smaller improvements and are accompanied by increased language-wise segregation. These results support a causal role for language discrimination in reducing the additional cost of multilingual learning.