Speech recognition improves for bilingual online Spanish classrooms

Beyond Aggregate WER: In-domain Adaptation, Learner Speech, Code-Switching, and Speaker Attribution in Online Spanish Learning Classrooms

Computation and Language

Summary

Transcribing conversations in online Spanish learning classes is tricky because learners switch between Spanish and English and make different kinds of speech mistakes. The authors tested several speech recognition systems and found that adapting models with classroom-specific data helped capture these conversations better than using much larger out-of-domain datasets. Their adapted system identified speech errors more accurately across different speakers and language mixes. This work shows that training with data similar to the target environment is important for good transcription in bilingual educational settings.

What this means in practice

  • For language technology developers: Improve ASR models for bilingual classroom conversations by adapting to interactional and multilingual speech features using small domain-matched datasets.
  • For call center analytics teams: Refine speaker diarization and transcription accuracy in multilingual customer service calls with tailored adaptation on in-domain interaction data.

Authors

Olga Kellert, Samhitha Harish, Johnatan E Bonilla, Nemika Tyagi, Phanos Kareen, Nelvin Licona-Guevara

Abstract

Automatic speech recognition (ASR) increasingly supplies transcripts for educational NLP, but aggregate word error rate (WER) can conceal errors concentrated in learner speech, multilingual turns, and speaker attribution. We compare platform-integrated, open-weight, adapted, and commercial ASR configurations on held-out sessions of online L2 Spanish interaction between L1 English learners and their instructors and tutors. Because English is the learner's shared first language, the interactions contain Spanish-only, English-only, and code-switched turns. Evaluation combines overall WER and character error rate (CER), WER disaggregated by speaker role and language context, and diarization error rate (DER). We also adapt Whisper large-v3 and MOSS-Transcribe-Diarize with LoRA using human-corrected in-domain classroom data. MOSS adaptation produced consistent improvements across speaker roles and language contexts, whereas Whisper showed smaller gains. Domain match proved more important than data volume: Larger out-of-domain bilingual and learner corpora did not outperform the smaller classroom corpus, where learner speech and code-switching are embedded in instructor--learner interaction. The open-weight MOSS adapter was competitive with commercial Muse Voice Transcribe on WER and CER and achieved lower system-provided DER on the recordings eligible for each system. Learner, English-only, and code-switched speech remained more difficult than instructor and Spanish-only speech. The findings demonstrate the value of condition-aware evaluation and interactionally matched data for bilingual educational ASR.
View PDFOpen arXiv