Pixel language models improve cross-language text understanding with scale
Scaling Multilingual Pretraining of Pixel Language Models
Computation and LanguageArtificial Intelligence
Summary
Pixel language models look at text as images instead of using word lists, which helps them work well across many languages. The authors studied how making these models bigger and training them on more languages and scripts changes their abilities. They found that bigger models help especially with the languages and writing systems seen during training. Models also get better with harder languages that don’t have many examples, beating other common methods. However, adding more languages can reduce how well models align different languages internally.
What this means in practice
- •For machine translation engineers: Build multilingual text understanding systems that work well on rare languages and diverse scripts using pixel-based models.
- •For natural language processing developers: Create robust text processing tools that generalize across many languages without relying on fixed vocabularies or tokenizers.
Authors
Ilker Kesen, Jenny Kunz, Kasra Malihi, Desmond Elliott
Abstract
Pixel language models process text as rendered images instead of a finite vocabulary, which gives them strong cross-lingual transfer abilities. However, not much is known about how these abilities are influenced by larger-scale pretraining. In this paper, we train a family of MPIXEL models along three axes of variation: more languages, more scripts, and larger model capacity under fixed per-language compute. In evaluations on five tasks covering 83 languages, we find that scaling acts locally: each axis improves the added languages and scripts, while increased capacity mitigates the curse of multilinguality from pretraining on more languages. Transfer to unseen scripts improves with multilingual pretraining, but does not further improve with scaling. On typologically distant low-resource languages, our models outperform subword and character-level baselines across many writing systems. We also show that larger pixel models reserve more layers for visual processing, and that cross-lingual alignment weakens as more languages are added. Overall, our findings characterize how multilingual scaling improves pixel language models.