Language models learn differently from tokens as targets and context
Decoupling Token Roles in Autoregressive Pretraining
Machine LearningArtificial IntelligenceComputation and Language
Summary
Predicting the next word is key to training language models, but each word also helps shape what comes next. The authors show that fixing a word to be easier to guess makes it less harmful as a target but more influential as context. This means models learn differently from a word when predicting it versus when using it as context. Understanding these two roles separately can improve how we design and analyze language models.
What this means in practice
- •For language model engineers: Improve training techniques by separately optimizing how tokens contribute as prediction targets and as context to reduce errors in generated text.
- •For ai content generation teams: Analyze generated text quality by understanding how chosen tokens influence both next-token likelihood and the context for future tokens.
Authors
Suqin Yuan, Runqi Lin, Kevin Qinghong Lin, Junchi Yu, Lei Feng, Chris Russell, Tongliang Liu
Abstract
Autoregressive pretraining increasingly draws on heterogeneous data, making it important to understand how a model learns from an individual token. The next-token prediction objective naturally identifies a token's contribution with its own loss. However, each token is not only a prediction target but also context for what follows. Using controlled corruption, we decouple these two roles and find a reversal: making a noisy token easier to predict reduces its damage as a target but increases it as context. The same decoupling helps explain text generated by language models: generation selects each token by its fit to the prefix, while its role as context is never tested against an independently determined continuation, because that continuation is generated to fit it. At known corrupted positions, acting through the context can reduce damage that removing the token's own loss does not. Understanding and controlling what a model learns from a token therefore requires decoupling its roles.