Speech recognition accuracy drops significantly in southeast asian languages
SEA-HELM Audio: Evaluating Multilingual ASR-LLM Pipeline Performance Across SEA-HELM Subsets
SoundComputation and Language
Summary
Speech recognition systems combined with language understanding models work less accurately when dealing with spoken input instead of text, especially in Southeast Asian languages. The researchers created SEA-HELM Audio, a test that checks how much performance drops for five languages including Tamil, Malay, and Chinese. They found big declines in correctly answering questions and following instructions, with Tamil suffering the most. This shows that just testing on text isn’t enough to know how reliable voice assistants will be in these languages.
What this means in practice
- •For voice assistant developers: Test and improve multilingual voice assistants by measuring how speech recognition errors impact language model reasoning and instruction tasks.
- •For customer support platform teams: Evaluate speech-to-text pipelines across Southeast Asian languages to ensure reliable automated handling of spoken customer queries.
- •For speech recognition product teams: Design more accurate multilingual ASR systems by identifying which languages have the largest performance drops when paired with language models.$Commercial implications: Enables development of new multilingual speech recognition products that better support Southeast Asian languages, improving voice assistant performance.
Authors
Yixian Tan, Aulia Adila, Jonathan Heng, Kunat Pipatanakul, Donghang Wu, Anocha Sutaveephamochanon
Abstract
We introduce SEA-HELM Audio, a Southeast Asian (SEA) multilingual evaluation framework that extends the text-based SEA-HELM benchmark to assess performance degradation in cascaded automatic speech recognition and large language model (ASR-LLM) pipelines. Using synthesized speech, we compare speech-input and text-input performance across five languages (Chinese, English, Malay, Indonesian, and Tamil) on multiple-choice reasoning and instruction-following tasks. We evaluate open-weight and commercial ASR-LLM systems using identical verifiers and quantify the speech gap as the difference between text and speech accuracy. Results show substantial degradation, with accuracy drops of 9.00 to 18.50 percentage points for multiple-choice reasoning and 20.00 to 42.86 points for instruction following. Tamil exhibits the largest degradation, exceeding 40 percentage points. Our findings highlight the limitations of text-only benchmarks in predicting voice-agent reliability and demonstrate the value of SEA-HELM Audio for evaluating multilingual voice-agent systems.