DeepSEA benchmark shows language models struggle with Southeast Asia languages
DeepSEA: A Multilingual Evaluation Benchmark for Southeast Asia
Computation and Language
Summary
Many AI language models perform well with popular languages but have trouble with less common Southeast Asian languages. The authors created DeepSEA, a new test set covering 12 languages in Southeast Asia, including some rarely supported ones. They tested various AI models and found that smaller models especially struggle with these lower-resource languages, while the best models do better but still rely heavily on English. This work helps highlight challenges in making AI more fair and capable across diverse languages.
What this means in practice
- •For multilingual model developers: Evaluate and improve AI models on diverse Southeast Asian languages with a standardized, human-translated multilingual benchmark.
- •For translation technology teams: Test machine translation quality on under-represented Southeast Asian languages to identify weaknesses and target improvements.
Authors
Jiuzhou Han, Abhishek Bijay Mishra, Trevor Cohn, Shubham Mittal
Abstract
While language models have demonstrated remarkable capabilities, their performance remains disproportionately skewed towards high-resource languages. This disparity is particularly acute for the 700 million inhabitants of Southeast Asia, where vast linguistic diversity remains under-represented in existing benchmarks, often suffering from limited language coverage or narrow task scopes. To address this, we introduce DeepSEA, a comprehensive benchmark constructed from established datasets through professional human translation into 12 Southeast Asian languages, spanning both high-resource (e.g., Indonesian, Thai, Vietnamese) and low-resource (e.g., Burmese, Khmer, Lao) tiers across five distinct tasks: mathematical reasoning, scientific reasoning, question answering, abstractive summarization, and machine translation. Extensive evaluation of frontier proprietary and open-weights models reveals that while frontier models like Gemini-3.7-Flash maintain robustness across resource tiers, smaller models face severe performance degradation on low-resource languages. Crucially, models perform better in reasoning and summarization when pivoted through English rather than target languages, exposing a critical dependency on English representations, especially for low-resource languages. Beyond these observations, we provide a cross-lingual error analysis and a cross-script tokenizer fertility analysis. We hope these findings shed light on the development of next-generation inclusive multilingual models.