Search method shapes tokeniser performance more than optimization goal
Objective vs. Search: Decomposing What Makes a Good Tokeniser
Computation and LanguageArtificial Intelligence
Summary
Tokenisers break text into pieces for language models, using different methods to decide these pieces. Two popular methods differ in their aims and how they find the best pieces, but previous studies mingled these differences. The authors created new tokenisers to separate the effects of goal and search technique and found that how the search is done matters most for model efficiency. However, when testing language understanding, no clear advantage was linked to either approach. This work helps builders design tokenisers more thoughtfully.
What this means in practice
- •For language model engineers: Choose bottom-up search tokenisers to improve language model compression and efficiency across vocabulary sizes and domains.
- •For multilingual nlp teams: Design tokenisers with bottom-up search to achieve better data compression when building multilingual language models.
Authors
Ahmetcan Yavuz, Clara Meister, Tiago Pimentel
Abstract
Two dominant tokenisation algorithms are used by modern language models: byte-pair encoding (BPE) and UnigramLM. These differ along two orthogonal axes: their optimisation objective (compression vs. log-likelihood) and their search procedure (bottom-up merging vs. top-down pruning). Existing comparisons confound these axes, making it unclear whether their observed differences stem from what is being optimised vs. how it is being optimised. We disentangle the two by introducing two new tokenisation algorithms that complete this 2x2 design space: BottomUpLL, a bottom-up likelihood-based tokeniser, and TopDownComp, a top-down compression-based tokeniser. We train language models with tokenisers produced by each algorithm, varying: model size, vocabulary sizes, and domain (English-only vs. multilingual). Evaluating models on bits-per-byte, we find that the search procedure -- not the objective -- is the dominant factor: bottom-up tokenisers consistently achieve lower bits-per-byte in most settings. Evaluating models on the BLiMP task, however, shows no consistent relationship between design choice and performance. Overall, our results disentangle the effect of tokeniser design choices on language modelling performance, offering concrete guidance for their more principled construction.