Latent core tokenizer improves language tokenization across 104 languages
Latent Core Tokenizer: Compress, but Meaningfully
Computation and Language
Summary
Tokenizers break down text into smaller pieces called tokens, which help computers understand language. The authors found that simply making token vocabularies smaller does not make them better for all languages. They created the Latent Core Tokenizer (LCT), which finds meaningful language building blocks before making a shared vocabulary for many languages. This method works better than other tokenizers on many languages and tasks, showing that understanding word structures is important, not just making the vocabulary compact.
What this means in practice
- •For multilingual model engineers: Build token vocabularies that improve language representation quality across diverse languages using morphology-aware structural discovery.
- •For natural language processing platform developers: Integrate advanced tokenization techniques to boost multilingual benchmark performance with meaningful token units identified by LCT.
- •For language technology product teams: Deliver better multilingual applications by applying LCT to create balanced vocabulary resources that enhance coverage across languages.$Commercial implications: Enables improved multilingual NLP products by providing vocabulary optimized for diverse linguistic structures, benefiting global users.
Authors
Felermino D. M. A. Ali, Millicent Ochieng, Ogbemi Ekwejunor-Etchie, Ade Famoti, Jacki O'Neill, Debjit Paul
Abstract
Tokenizers are commonly optimized for compression, but a compact vocabulary does not necessarily distribute its capacity evenly across languages. We introduce the Latent Core Tokenizer (LCT), a language-agnostic approach that separates structural discovery from vocabulary construction. LCT uses Minimum Description Length, entropy-based boundary signals, and morphotactic constraints to identify reusable linguistic units before constructing a shared vocabulary. Across 104 languages with a 200K-token vocabulary, LCT achieves lower fertility and higher MorphScore than BPE, Unigram, and parity-aware BPE, while maintaining comparable cross-lingual disparity in tokenization cost. Across four multilingual downstream benchmarks, LCT improves aggregate score by 1.48, 1.83, and 2.00 points over BPE, Unigram, and parity-aware BPE, respectively. Our findings show that compression alone does not predict representation quality and highlight the importance of morphology-driven structural discovery and how frequency is used to allocate the final vocabulary across languages.