Smooth Scaling Laws Hide Stepwise Token Learning
2026-06-29 • Computation and Language
Computation and Language
AI summaryⓘ
The authors studied why large language models get better in a predictable way as they get bigger or see more data. They looked closely at how individual words (tokens) are learned over time during training and found that learning happens in quick bursts. By measuring when each token becomes easier to predict, they showed that this timing pattern explains the overall improvement trends (scaling laws) in the models. They also used this insight to change the training process and made the model learn faster by 11%. This work shows that understanding token-level learning helps explain and improve how language models get better.
language modelscaling lawstoken-level learningloss trajectoryvalidation losstraining distributionmodel scalingdata scalingoptimization trajectorylearning-time spectrum
Authors
Pingjie Wang, Zechen Hu, Peiru Yang, Fu Guo, Debing Zhang
Abstract
Language model loss follows remarkably regular scaling laws over model and data size, yet it remains unclear why the aggregate loss should exhibit a power-law form. Existing explanations often attribute this regularity to a heavy-tailed spectrum of pattern difficulty in natural language, but this view has not been directly validated at token-level granularity in large-scale real-data training. We present a token-level framework that decomposes scaling laws into localized learning events of individual contextualized tokens. By fitting token loss trajectories with sigmoids, we show that token learning is concentrated in localized transitions, giving rise to a learning-time spectrum that dominates the scaling-law shape. Across more than one hundred pre-training runs on large and diverse real-language corpora with modern LLM architectures, scaling up to 6B parameters and 300B training tokens, the measured learning-time spectrum quantitatively reconstructs the validation loss derivative along the training-step $T$, data-scale $D$, and model-scale $M$ axes. We further show that the same signal is actionable: by reshaping the training distribution according to when tokens become learnable, we alter the optimization trajectory and achieve 11\% faster validation-loss reduction. These results provide direct empirical evidence that scaling laws are governed primarily by the distribution of token-level learning times, and that this distribution can be used not only to explain scaling behavior but also to improve training performance.