Adaptive tokenization improves language models for mixed text and code
Adaptive Frequency Threshold Byte Pair Encoding for Mixed Natural Language and Structured Symbolic Corpora
Computation and LanguageMachine Learning
Summary
Language models often split text into smaller pieces called tokens to understand and generate language. Common methods struggle when important phrases span multiple words or include special symbols, like in math notation or computer code. The authors propose AdaBPE, a new way to split text that changes its rules based on the type and length of the tokens, allowing better handling of specialized formats like LaTeX alongside regular language. This method uses fewer tokens and makes language models work better with mixed content without losing performance on normal text.
What this means in practice
- •For natural language processing engineers: Improve language model tokenization to better handle documents combining text and structured code like LaTeX with more efficient encoding.
Authors
Pietro Di Nicco, Lucio La Cava, Andrea Tagarelli
Abstract
Modern language models rely on subwording tokenization to define the discrete units of input processing, typically using methods like Byte Pair Encoding (BPE) that treat whitespace as a fixed semantic boundary. This design is inherently limited, as many meaningful concepts---including multiword expressions and the structured environments of formal languages like LaTeX---frequently span across whitespaces. While recent superword tokenization methods attempt to address this by relaxing whitespace constraints, they often rely on greedy frequency-driven merging that fails to account for token length or the specific needs of underrepresented domain-specific languages. In this paper, we introduce Adaptive Frequency Threshold Byte Pair Encoding (AdaBPE), a new tokenization method that leverages an adaptive frequent-pattern mining approach. AdaBPE features a twofold dynamicity: it adapts minimum frequency thresholds based on both the length of a candidate token, to prevent favoring frequent short patterns, and its type, to distinguish between general and formal language patterns, using LaTeX as case in point. This allows the model to efficiently allocate vocabulary capacity to domain-specific notations, such as LaTeX, even when they represent a small fraction of the training data. We evaluate AdaBPE across five datasets spanning LaTeX and natural language, showing consistent gains in encoding efficiency and downstream language-model performance on domain-specific content without compromising general-language performance.