Papers for

translation technology teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

DeepSEA benchmark shows language models struggle with Southeast Asia languages

DeepSEA: A Multilingual Evaluation Benchmark for Southeast Asia

Abstract: While language models have demonstrated remarkable capabilities, their performance remains disproportionately skewed towards high-resource languages. This disparity is particularly acute for the 700 million inhabitants of Southeast Asia, where vast linguistic diversity remains under-represented in existing benchmarks, often suffering from limited language coverage or narrow task scopes. To address this, we introduce DeepSEA, a comprehensive benchmark constructed from established datasets through professional human translation into 12 Southeast Asian languages, spanning both high-resource (e.g., Indonesian, Thai, Vietnamese) and low-resource (e.g., Burmese, Khmer, Lao) tiers across five distinct tasks: mathematical reasoning, scientific reasoning, question answering, abstractive summarization, and machine translation. Extensive evaluation of frontier proprietary and open-weights models reveals that while frontier models like Gemini-3.7-Flash maintain robustness across resource tiers, smaller models face severe performance degradation on low-resource languages. Crucially, models perform better in reasoning and summarization when pivoted through English rather than target languages, exposing a critical dependency on English representations, especially for low-resource languages. Beyond these observations, we provide a cross-lingual error analysis and a cross-script tokenizer fertility analysis. We hope these findings shed light on the development of next-generation inclusive multilingual models.

Mon 19 OctComputation and Language
The gist
Many AI language models perform well with popular languages but have trouble with less common Southeast Asian languages. The authors created DeepSEA, a new test set covering 12 languages in Southeast Asia, including some rarely supported ones. They tested various AI models and found that smaller models especially struggle with these lower-resource languages, while the best models do better but still rely heavily on English. This work helps highlight challenges in making AI more fair and capable across diverse languages.
Open → 2610.26155v1