Papers for

multilingual speech technology product managers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Multilingual benchmark assesses speech recognition for banking calls

mu-bench: A Multilingual Utterance Transcription Benchmark

Abstract: Voice agents depend on accurate automatic speech recognition (ASR) to act on what callers say, yet ASR is evaluated on read, English-centric speech with word error rate (WER), which penalizes surface rather than semantic differences. We introduce mu-bench, a dataset of 4,270 caller utterances from 250 phone calls to an AI banking agent in English, Spanish, Turkish, Vietnamese, and Mandarin, centered on form-field inputs such as names, email addresses, and confirmation codes. We release Utterance Error Rate (UER), an LLM judge of whether a transcript preserves meaning, calibrated against human raters, together with an LLM normalizer that makes WER comparable across providers' output formats. On 1,847 human-rated transcripts, UER agrees with annotators at $κ$ = 0.78, versus 0.53 for exact-match WER on normalized text. We rank six commercial providers on a public leaderboard; the best reaches 11.9% UER, and Mandarin is hardest for all six.

Fri 25 SeptComputation and Language

The gist

Speech recognition systems often struggle to accurately understand what people say, especially in non-English languages and in conversational settings like phone calls. The authors created mu-bench, a dataset of real customer calls in five languages to test how well speech recognition systems can capture important information like names and codes. They also developed a new way to measure if the transcriptions keep the original meaning, not just count word mistakes. Their method agrees better with human judgments than traditional measures and shows Mandarin is the hardest language for current systems.

Open → 2609.32082v1