Language models learn rare knowledge faster later in training
What Improves During Pretraining? Tracking Knowledge Acquisition with Point-in-Time Models
Artificial Intelligence
Summary
Language models learn different types of knowledge from internet text collected over several years, but it was unclear how they learn these kinds of knowledge over time during training. The authors studied models trained on web data from different years to measure how consistently knowledge appears and how learning changes as training progresses. They found that knowledge that varies more over time starts out learned at a similar speed but improves faster later in training compared to more consistent knowledge. This means the model becomes better at learning less common or time-sensitive knowledge as it trains longer.
What this means in practice
- •For natural language processing teams: Identify which knowledge items benefit most from extended training to improve model performance on knowledge-heavy tasks over time.
- •For knowledge management software developers: Design better update schedules for language models by tracking fluctuations in knowledge prevalence across time-sensitive datasets.
Authors
Chang Su, Bin Wu, Emine Yilmaz, Yao Lu
Abstract
Language models pretrained on web corpora collected over multiple years learn both general and domain-specific knowledge. How these different kinds of knowledge are learned as pretraining progresses remains unclear. Recent point-in-time models trained on web snapshots from different years provide a way to probe how knowledge prevalence varies over time. Using these models, we propose a temporal variability score (TVS), defined as the coefficient of variation of an item's negative log-likelihood across models. A higher score suggests that the knowledge may be less consistently represented across years of web text. We evaluate six benchmarks covering diverse topics using six models from multiple open-source model families. We find that low- and high-TVS items are learned similarly early in training, but high-TVS items improve faster later on. At the end of the pretraining run, high-TVS items outperform low-TVS items in 34 of 36 model-benchmark settings, with an average advantage of 7.3 points of chance-normalized accuracy. This pattern is most robust on knowledge-intensive tasks, even when comparing items of similar difficulty. Our findings reveal differences in how knowledge is learned over the course of pretraining, with temporal variability providing a diagnostic of these learning dynamics beyond aggregate benchmark scores.