Nested Byte-Level Vocabularies Are Cheap to Deploy and Expensive to Share: A Pre-Registered Negative Result
Computation and LanguageArtificial IntelligenceInformation Retrieval
Summary
The authors studied a special tokenizer that works with different vocabulary sizes by using parts of the same rule list, enabling a language model to switch sizes without retraining. They trained multiple models and showed that slicing parts of the model's embeddings and outputs matches the full model's behavior exactly while using fewer weights and not slowing down. However, models trained to handle multiple vocabulary sizes were slightly less efficient than those trained for just one size, though they were more robust to typing errors. The authors suggest that training on multiple sizes improves error handling mainly because of the training itself, not the special control tokens used to switch sizes.
byte-level BPE tokenizervocabulary sizelanguage modelembedding slicingmulti-cap trainingoutput headlogitstypographical noisecontrol tokenmodel robustness
Authors
Christos Koutsiaris
Abstract
A byte-level BPE tokenizer is an ordered list of merge rules, so applying only a prefix yields a vocabulary whose token identifiers are the first rows of the full vocabulary. This prefix nesting allows one language model to operate at several vocabulary sizes, use a control token to indicate the active size, and be deployed at any trained size by slicing its embedding and output head. We pre-registered five claims, including margins, seeds, contrasts, and a stop rule, and trained 30 models with 3.1M- and 10.6M-parameter bodies on 200M tokens each. Slicing is numerically exact: across 76 checks, a sliced model reproduces the restricted full model's logits bit for bit and removes 66% of deployed weights without changing latency. However, the shared model trails a fixed-cap specialist by 3.64% bits per byte at 32k against a 1% margin, and by 2.96% at 8k against a 2% margin. A 2x2 ablation separating the control token from output restriction finds that the token changes performance by +0.07% to +0.13%, with all intervals crossing zero, while output restriction costs +0.47% to +1.19%; the factors are substitutes rather than complements. Multi-cap training nevertheless improves robustness: under typographical noise, the same checkpoint degrades 12.5--15.4 points less in its fine mode and outperforms each fixed-cap specialist at that specialist's vocabulary size. A control with neither cap token nor output restriction is equally robust, attributing this benefit to multi-granularity training rather than conditioning. The per-cap penalty tracks each cap's share of training rows, yielding a falsifiable prediction for future work.