TokEval: A Tokenizer Evaluation Suite
2026-08-18 • Computation and Language
Computation and LanguageMachine Learning
AI summaryⓘ
The authors created TokEval, a new way to measure how good tokenizers are for language models by looking at meaningful features beyond the usual counts, like how well they keep character boundaries or handle numbers. They tested different tokenizer designs by training language models under controlled conditions and then checked how well these models performed on tasks like language understanding, math, and code. Their results showed that some tokenizer features predict how well a model predicts language, while others relate to task accuracy. The authors suggest using TokEval to evaluate tokenizers more effectively without always needing extensive model retraining.
tokenizerlanguage model pretrainingbits-per-byteperplexityfertilitycompression rateUTF-8digit place-value boundarySpearman rhotask accuracy
Authors
Clara Meister
Abstract
Language model tokenizers are typically selected with minimal evaluation, despite the fact that their design choices directly impact model capabilities. This can be partly attributed to a limited understanding of which tokenizer properties affect which aspects of downstream performance. We introduce TokEval, a framework of tokenizer evaluation metrics that goes beyond standard measures like fertility and compression rate to capture linguistically and structurally meaningful properties, e.g., UTF-8 character boundary integrity and digit place-value boundary alignment for mathematics. To validate whether these metrics are predictive of downstream model performance, we conduct controlled language model pretraining experiments, varying solely the tokenizers' training data mixture, pretokenization strategy, and training algorithm. We evaluate the resulting models on bits-per-byte (a tokenizer-agnostic version of perplexity) and several benchmarks, spanning linguistic understanding, mathematical reasoning, and code generation. Our experiments suggest that different intrinsic properties have different impacts on model abilities: information-theoretic metrics predict language modeling abilities (Spearman rho up to 0.80), while structure-sensitive metrics, such as those measuring digit and line-break handling, correlate with task accuracy. We hope TokEval enables more principled tokenizer evaluation, replacing pretraining sweeps with intrinsic measurement wherever the two agree.