Multilingual benchmark assesses speech recognition for banking calls
mu-bench: A Multilingual Utterance Transcription Benchmark
Computation and Language
Summary
Speech recognition systems often struggle to accurately understand what people say, especially in non-English languages and in conversational settings like phone calls. The authors created mu-bench, a dataset of real customer calls in five languages to test how well speech recognition systems can capture important information like names and codes. They also developed a new way to measure if the transcriptions keep the original meaning, not just count word mistakes. Their method agrees better with human judgments than traditional measures and shows Mandarin is the hardest language for current systems.
What this means in practice
- •For voice assistant developers: Evaluate speech-to-text accuracy in multilingual customer interactions using a dataset and metric tuned for semantic correctness.
- •For customer support platform teams: Compare commercial speech recognition providers on meaningful transcription accuracy for banking call scenarios across five languages.
- •For multilingual speech technology product managers: Develop improved transcription products for banking calls by targeting the hardest language, Mandarin, identified via the benchmark.$Commercial implications: Enables creation of better speech transcription services for banking customers by focusing on challenging languages using this dataset and metric.
Authors
Andrea Li, Soham Ray
Abstract
Voice agents depend on accurate automatic speech recognition (ASR) to act on what callers say, yet ASR is evaluated on read, English-centric speech with word error rate (WER), which penalizes surface rather than semantic differences. We introduce mu-bench, a dataset of 4,270 caller utterances from 250 phone calls to an AI banking agent in English, Spanish, Turkish, Vietnamese, and Mandarin, centered on form-field inputs such as names, email addresses, and confirmation codes. We release Utterance Error Rate (UER), an LLM judge of whether a transcript preserves meaning, calibrated against human raters, together with an LLM normalizer that makes WER comparable across providers' output formats. On 1,847 human-rated transcripts, UER agrees with annotators at $κ$ = 0.78, versus 0.53 for exact-match WER on normalized text. We rank six commercial providers on a public leaderboard; the best reaches 11.9% UER, and Mandarin is hardest for all six.