Language models reduce repeated word steps with token building blocks
More Than Words: Compositional Tokenization for Efficient Language Models
Computation and Language
Summary
Language models normally predict one word or punctuation at a time, which makes the process slower. The authors propose a way to break down words into a base part and small add-ons, so fewer pieces are needed at each step. They trained models with this method and found they could shorten the text length by about 30% and improve performance a bit. This approach suggests language models can be more efficient by thinking of words as built from smaller reusable parts.
What this means in practice
- •For mobile app developers: Create text prediction apps that run faster by reducing the steps needed to generate phrases with compositional tokens.
- •For cloud service engineers: Improve efficiency of language model APIs by lowering computation cost through shorter token sequences using composed token representations.
Authors
Yuval Reif, Guy Kaplan, Roy Schwartz
Abstract
Language models process and generate text sequentially in token units, and the tokenizer determines how much text each inference step covers. Under standard tokenization, a short English phrase such as "On the table." is usually produced as four separate predictions for the preposition (On), article (the), noun (table), and punctuation (.), where each consumes a sequence position and adds inference cost. We introduce CoBPE, a compositional tokenization approach that represents such phrases as a lexical base token (table) attached with a small set of reusable surface modifiers, composed in embedding space at input and predicted jointly at output. In controlled pretraining from scratch at 780M and 1.3B scales, CoBPE shortens sequences by 30% and improves average downstream performance by 1.2 points relative to standard BPE under matched training compute. Our results suggest that part of what is now expressed through token sequences can instead be modeled through structured representations, opening a broad design space for more token-efficient and capable language models.