Papers for

multilingual model engineers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Latent core tokenizer improves language tokenization across 104 languages

Latent Core Tokenizer: Compress, but Meaningfully

Abstract: Tokenizers are commonly optimized for compression, but a compact vocabulary does not necessarily distribute its capacity evenly across languages. We introduce the Latent Core Tokenizer (LCT), a language-agnostic approach that separates structural discovery from vocabulary construction. LCT uses Minimum Description Length, entropy-based boundary signals, and morphotactic constraints to identify reusable linguistic units before constructing a shared vocabulary. Across 104 languages with a 200K-token vocabulary, LCT achieves lower fertility and higher MorphScore than BPE, Unigram, and parity-aware BPE, while maintaining comparable cross-lingual disparity in tokenization cost. Across four multilingual downstream benchmarks, LCT improves aggregate score by 1.48, 1.83, and 2.00 points over BPE, Unigram, and parity-aware BPE, respectively. Our findings show that compression alone does not predict representation quality and highlight the importance of morphology-driven structural discovery and how frequency is used to allocate the final vocabulary across languages.

Thu 8 OctComputation and Language

The gist

Tokenizers break down text into smaller pieces called tokens, which help computers understand language. The authors found that simply making token vocabularies smaller does not make them better for all languages. They created the Latent Core Tokenizer (LCT), which finds meaningful language building blocks before making a shared vocabulary for many languages. This method works better than other tokenizers on many languages and tasks, showing that understanding word structures is important, not just making the vocabulary compact.

Open → 2610.12376v1