Byte based models exceed token models after more training
Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
Training small language models is often done by copying knowledge from larger models using a method called distillation. This paper compares models that work with normal words (tokens) versus those that use bytes, the smallest unit of digital text. The authors found that, although token-based models start off better when trained with limited computing power, byte-based models improve more steadily and eventually outperform token models with more training. Byte models also need less data and memory during training and prediction. This suggests that using bytes could make smaller models stronger and more efficient.
What this means in practice
- •For machine learning engineers: Build smaller language models that achieve better accuracy with less data and memory using byte-level tokenization and distillation.
- •For software developers: Develop applications processing natural language more efficiently by deploying byte-based models requiring less storage for predictions.
Authors
Kalyani Marathe, Artidoro Pagnoni, Tomasz Limisiewicz, Margaret Li, Mike Lewis, Luke Zettlemoyer, Srinivasan Iyer
Abstract
Small models are made more capable through distillation from a larger one that shares their tokenization scheme. However, do distilled byte and token models behave similarly in terms of scaling trends as compute and data increases? To enable this comparison, we introduce two variants to efficiently convert token logits to Byte Logits: 1) approximate: Marginalize-It, and 2) exact: End-Of-Token. We then present the first large scale study of overtraining decoder-only dense transformer models varying two dimensions simultaneously: the tokenization scheme (Tokens, Bytes, Bytes w/ eot) and the training objective (Distillation vs. Cross-Entropy), sweeping layer-parameter-matched models with roughly 1 billion parameters up to 1 trillion bytes of data. Across eight benchmarks spanning three categories: Multiple Choice QA, Language Generation, and Machine Translation, we find that Token-1B models outperform byte models (End-Of-Token-1B and Bytes-1B) in the low-FLOP regime but eventually plateau; byte models start worse yet surpass Token-1B models with more compute, reaching a higher downstream task performance ceiling. Extrapolating the average top-1 error vs. validation BPB scaling laws predicts that, asymptotically, distilled End-Of-Token-1B outperforms distilled Token-1B by up to 4%. They are also far more data efficient, matching the performance of distilled Token-1B using only one-sixth of the training data. Moreover, by operating over a small vocabulary of 256 bytes instead of on the order of 100K tokens, they circumvent the need for top-k truncation during logit dumping, while also reducing logit storage costs to roughly one-fifth. Finally, our downstream performance scaling laws predict that our distilled End-Of-Token-1B models asymptotically surpass the Llama 3.2-1B, Gemma-3-1B-pt, and Gemma 2B models on averaged downstream tasks by up to 6.5%, 8.1%, and 2.1%, respectively.