Audio models reveal hidden errors in code switched English Yoruba speech

Beyond Word Error Rate: A Switch Aware Evaluation of ASR and Audio Language Models on English Yoruba Code-Switched Speech

Computation and LanguageArtificial Intelligence

Summary

Speech recognition systems usually perform well on single languages, but this paper shows that when people mix English and Yoruba in the same sentence, the usual error scores don’t tell the full story. The authors tested eleven modern speech systems on mixed English-Yoruba speech and found that while overall error rates look good, these systems struggle to recognize Yoruba words correctly and make many mistakes at language switching points. Some systems even add extra words or change the meaning depending on how the test questions are framed. The authors provide tools to better measure these mistakes in mixed language speech.

What this means in practice

  • For speech technology developers: Develop more accurate speech recognition tools by using switch-aware metrics to detect errors specifically at points where languages change in mixed speech.
  • For voice assistant engineers: Improve mixed language understanding in voice interfaces by using refined evaluation methods that identify failures in recognizing low-resource language words and language switches.

Authors

Chibuzor Okocha, Christan Earl Grant

Abstract

Automatic speech recognition (ASR) systems and audio language models (audio LMs) now report low error rates on monolingual benchmarks, but their behavior on code switched speech in low resource, diacritic rich languages remains poorly characterized. We present a switch aware evaluation of eleven modern systems (six ASR models and five audio LMs) on English Yoruba code-switched speech, using a deterministic 2000 utterance evaluation set and a shared scoring pipeline. Beyond word error rate (WER), we report switch localized diagnostics: a switch entry token error rate (SETER), windowed switch point error rates, language specific error rates, and a diacritic insensitive WER. Our central finding is that aggregate WER hides code switching behavior. The best system by WER (an ASR model) is statistically indistinguishable from a leading audio LM on WER, yet the audio LM is significantly better on every switch localized metric. Across faithful systems, Yoruba token recognition collapses (error 0.97 for almost all systems) while English tokens are recognized far better, and errors concentrate sharply at switches into Yoruba. Several generative audio LMs fail as exact transcribers, producing translation, verbosity, and prompt leakage that are strongly prompt dependent. We release manifests, metric implementations, and evaluation scripts to support reproducible, switch aware benchmarking for African code switched speech.
View PDFOpen arXiv