Code-switching language ID predicts multiple spoken languages in utterances
Code-Switching Spoken Language Identification as Multi-Label Set Prediction
Computation and Language
Summary
Usually, computers try to figure out what language someone is speaking, but they get confused when the speaker switches between languages in the same sentence. This paper looks at how to teach computers to detect all the languages used in a mixed speech part together, rather than just guessing one. The researchers made a new system that predicts the full set of languages without needing to know how many languages appear beforehand. Their approach works well but still isn’t perfect because of challenges like unbalanced training data and differences between made-up and real speech.
What this means in practice
- •For speech technology developers: Improve language recognition systems to detect multiple languages spoken in a single utterance, enhancing code-switching speech processing.
- •For call center software engineers: Build better call routing and transcription by identifying all languages used in mixed-language customer calls automatically.
Authors
Shunsuke Mitsumori, Matthew Wiesner, Shigeo Morishima, Shinji Watanabe
Abstract
Code-switched (CS) speech leaks through the monolingual language identification (LID) filters used to curate massive speech corpora, calling for CS-aware LID (CS-LID). We formulate utterance-level CS-LID as multi-label language-set prediction and propose a set generator that directly outputs the languages in an utterance, comparing it against atomic-pair and score-based classification baselines. Oracle Top-k is the strongest baseline, but thresholding fails because no single threshold separates CS from monolingual speech. Our set generator predicts the correct language count on unseen pairs without assuming the number of languages, but underperforms oracle Top-k in exact set accuracy. Our analysis identifies the key obstacles to robust CS-LID: oracle cardinality, threshold instability, language bias in CS training data, and the synthetic-to-real gap.