Improving language model tokenizers with smarter vocabulary choices
Rethinking BPE: Merge Selection as Model Selection
Computation and Language
Summary
Language models break text into smaller pieces called tokens, and choosing the right set of tokens is important for balancing model size and performance. The authors look at this process as a model selection problem, using statistics from the data to pick the best tokens instead of relying on a fixed arbitrary limit. They also create extra data to help when there isn’t much original text available. Their experiments show these methods reduce model size without hurting accuracy, and help especially when data is scarce.
What this means in practice
- •For natural language processing teams: Build smaller language models by selecting token vocabularies that maintain performance while using fewer computing resources.
- •For mobile app developers: Create efficient on-device language models for apps with limited data and memory by using improved tokenization techniques.
Authors
Vijini Liyanage, François Yvon
Abstract
Subword tokenization methods such as Byte-Pair Encoding (BPE) are a core component of modern language models; yet, tokenizer capacity remains typically determined by a fixed merge budget, chosen heuristically. Since larger vocabularies increase memory usage and computational cost, selecting compact and effective token inventories is essential for efficient language modeling. We study the optimization of a BPE merge budget as a statistical model selection problem and investigate information-theoretic criteria for \emph{computing optimal merge sets} directly from corpus statistics. To further improve the merge selection in limited data conditions, we augment the BPE training corpora with synthetic texts generated from the original data, providing estimates for the unigram and bigram distributions. Experiments across multiple corpora and downstream tasks show that using a principled vocabulary selection technique reduces model size, with little degradation in performance. We further observe that synthetic data can yield better tokenizers, especially in low-resource settings.