Multilingual language models gain speed and fairness with modular tokenizers
To Each Language Its Tokenizer: Modular Tokenizers for Efficient Multilingual LLMs
Computation and Language
Summary
Multilingual language models often use a single shared vocabulary for all languages, which can make them less efficient or fair for some languages. The authors developed a method that creates modular tokenizers—special word break-up tools—that can be customized for different languages or groups of languages. This approach reduces memory use and speeds up language processing without losing quality, making models fairer and more efficient when handling multiple languages. The training method also helps the model learn efficiently with these separate vocabularies.
What this means in practice
- •For multilingual chatbot developers: Build chatbots that customize tokenization to only used languages, reducing memory needs and speeding up responses with no loss in accuracy.
- •For mobile app developers: Create language apps that run faster and use less memory on devices by loading only relevant language-token modules.
Authors
Franck Signe, Hippolyte Pilchen, François Yvon, Édouard Grave
Abstract
Multilingual Large Language Models (LLMs) traditionally rely on a single vocabulary shared by all supported languages, which can lead to uneven compression across them. Moreover, their large embedding and output matrices increase memory usage and slow inference, notably for small-scale models. It is also wasteful as models are often used for only a subset of languages. To address these issues, we introduce a modular framework for multilingual model training. First, we propose methods to learn large modular BPE and Unigram tokenizers that enable extraction of subtokenizers tailored to any language subset. These subtokenizers achieve compression on par with monolingual tokenizers and improve cross-lingual fairness. Second, we design a pretraining strategy that samples subtokenizers to form batches, restricting predictions to the relevant vocabulary subset and allowing efficient training despite a large vocabulary. This supports efficient inference with any combination of language-specific vocabularies. Therefore, it reduces memory usage and speeds up inference in models without sacrificing performance.