Text encoding affects language model accuracy for some languages
When does text encoding matter for language modeling?
Computation and Language
Summary
This paper looks at how the way text is encoded affects language models, the AI tools that understand and generate language. It shows that for languages like Chinese and Korean with characters that pack a lot of information, choosing better encoding methods can improve model performance. But for languages like Georgian and Tamil, which have simpler characters, the encoding doesn't make much difference. The authors suggest that text encoding should be chosen based on the language rather than using the same method for all languages.
What this means in practice
- •For machine learning engineers: Optimize language model performance by selecting encoding schemes tailored to specific languages, improving efficiency for complex-script languages.
- •For software localizers: Improve localization quality by adapting text encoding strategies to better suit the target language's script characteristics.
Authors
Eugene Jang, Catherine Arnett, Terra Blevins
Abstract
UTF-8's role as the de facto encoding standard in language models introduces cross-lingual inequities due to its variable-length design. However, the effect that this encoding choice has on model performance remains largely unexamined. In this work, we investigate how text encoding impacts the performance of byte-level language models, comparing UTF-8 with a wide range of other encoding strategies, including other Unicode-based approaches, language-specific encodings, and our own customized encodings. To facilitate our analysis, we propose Aether: a hybrid encoding scheme that uses script-specific encodings for our target setting while maintaining full Unicode coverage through UTF-8 fallbacks, using this framework to model four orthographically diverse languages. Surprisingly, we find that while languages with information-dense characters (Chinese, Korean) significantly benefit from compressive encodings, languages whose characters carry less information individually (Georgian, Tamil) are largely insensitive to encoding choice. In light of these findings, we argue that text encoding should be a language-specific design decision, rather than a universal default.