Language model training benefits from ensembling and careful stopping
A Scaling Theory of Multi-Epoch Pretraining: Late-Time Overfitting and Ensembling
Machine Learning
Summary
Training large language models involves showing them data many times (epochs) and using lots of computing power. The paper studies what happens when the total amount of data is fixed but models are trained multiple times over it. It finds that combining multiple trained models (ensembling) helps prevent performance from getting worse after too much training and allows training to continue longer. The authors use a mathematical approach to explain why ensembling reduces errors and how to best split computing resources between training longer or training more models.
What this means in practice
- •For machine learning engineers: Optimize compute use by balancing individual model training time and ensemble size when dataset size is fixed to improve language model generalization.
- •For natural language processing teams: Improve language model accuracy by leveraging ensembling with logit averaging to reduce late-stage overfitting on fixed data resources.
Tested on one dataset.
Authors
Zhen Yang, Yidi Miao, Blake Bordelon
Abstract
In the near future language model pretraining will become bound by total data rather than total compute. Such a regime requires revisiting scaling laws and scaling law theory. Motivated by prior works that have found utility in ensembling, we study how the number of training epochs and ensemble size jointly shape generalization when training data are fixed. In language-model experiments on FineWeb, logit averaging reduces late-training deterioration, delays optimal stopping, and can outperform wider or deeper single models at matched parameter-token budgets over a common training horizon. To analyze this interaction between data reuse and ensembling, we derive a dynamical mean-field theory (DMFT) for multi-epoch stochastic gradient descent in power-law random-feature ensembles, tracking error correlations across epochs and models. Ensembling reduces the prediction variance responsible for late-time growth in expected test error, changing the optimal stopping time. We use these stopping times to derive compute-allocation laws at fixed data and individual model sizes. Under this allocation, ensemble size grows with compute at least as fast as training time per member, while excess test loss above the limiting loss decreases as a power of compute.