Multilingual speech models overcome mistakes caused by mixed languages
Phonological Interference in Multilingual Speech Models
Computation and LanguageMachine Learning
Summary
Speech models that convert sounds to phonemes struggle when people speak in more than one language or use languages they weren't trained on. The authors found that these models wrongly assume the whole speech is in one language, causing errors called phonological interference. They measure how often the models miss sounds unique to one language when two languages mix. To fix this, they propose a new technique called windowed language estimation that adjusts the model’s language guess over small speech segments, reducing errors significantly without hurting performance on single-language speech.
What this means in practice
- •For speech recognition developers: Improve accuracy of phoneme-based speech recognizers on code-switched and low-resource languages by reducing phonological interference during inference.
- •For multilingual text-to-speech engineers: Enhance phoneme output fidelity in multilingual TTS systems for mixed-language input by using windowed language estimation to better track phonology shifts.
- •For language technology product managers: Create speech products that handle mixed-language input more reliably by integrating phonological interference correction techniques.$Commercial implications: Enables commercial speech applications with improved code-switching and low-resource language support, meeting user demand for seamless multilingual interaction.
Authors
Moran Yanuka, Raja Giryes, Moris Alper
Abstract
Phoneme-level models transcribe or generate speech as a sequence of phonemes, the smallest sound units that distinguish words. These models enable fine-grained pronunciation control and understanding, yet often fail on input that does not match any single training language, such as speech alternating between two languages, known as code-switching, or low-resource languages absent from training. We identify a systematic failure mode behind this, phonological interference: models assume the input is in a single language and impose its phonology, overriding local phoneme-level decisions that conflict with the assumed language. We measure interference by how often a model retains phonemes that one language has but the other lacks. On code-switched input, two phone recognizers (speech-to-phoneme models) and a phoneme-conditioned text-to-speech model lose 32% to 79% of these phonemes, but lose far fewer of the phonemes both languages share. On unseen languages, we find that phone recognizers impose the phonology of the training language they assign to the speech, and the more confident the assignment, the more they lose phonemes the unseen language has but the assigned language lacks. We probe the models' language estimate from their internal activations, and trace interference to a low dimensional subspace. On monolingual speech, steering this subspace toward another language makes the model lose the phonemes that only the original language uses and produce phonemes that only the target language has. We introduce windowed language estimation (WLE), an inference time repair that replaces the model's language estimate in this subspace with one computed from a short window around each position. On code-switched input, WLE removes 34% to 69% of the interference in all three models, and in the recognizers it leaves monolingual performance essentially unchanged.