Synthetic pretraining fails to beat natural language training alone
Rethinking Pre-pretraining
Computation and LanguageArtificial Intelligence
Summary
This paper finds that training language models first on computer-generated (synthetic) data before real language doesn’t improve performance compared to just training on real language data alone, even when adjusting the amount of training. The authors tested four types of synthetic data and multiple training setups, but none outperformed natural language training on its own. They also found that earlier reports of benefits from synthetic pretraining rely on specific ways of starting the training, and that models tend to forget what they learn from synthetic data when they later train on real language. These findings suggest that using synthetic data as a separate early step in training may not be helpful.
What this means in practice
- •For machine learning engineers: Allocate training compute directly to natural language data instead of synthetic data for improved language model performance under resource constraints.
- •For ai product managers: Make informed decisions about training strategies by prioritizing natural language datasets, avoiding synthetic pretraining stages that add complexity without benefit.
Authors
Alexander Stern, Tim Rocktäschel, Acyr Locatelli, Laura Ruis
Abstract
We report that allocating compute to synthetic pre-pretraining (PPT) never outperforms pure natural-language training in our compute-matched experiments. This holds across four PPT curricula (k-Dyck, neural cellular automata (NCA), declarative reference corpora, and code) and three training regimes: data-constrained, data-unconstrained, and a high token-to-parameter-ratio regime (881M parameters trained on 50B tokens). When natural language is scarce, repeated epochs over a 100M-token corpus beat every synthetic curriculum on natural-language validation loss, with no consistent downstream accuracy advantage from PPT. We find that PPT gains reported in prior work are sensitive to the weight initialization scheme. In an NCA reproduction, we recover a 5.4% perplexity improvement from PPT under the original initialization scheme. An alternative scheme improves both methods, but the no-PPT baseline more, closing the gap between them. However, when PPT receives additional compute rather than a share of the natural-language budget, gains persist at small natural-language budgets even with improved initialization, but disappear by 1B tokens. Finally, models forget what they learn during PPT in subsequent natural-language training; interleaving the same data throughout training prevents forgetting, but does not significantly improve natural-language performance. Our results suggest limited benefits of synthetic pre-pretraining as a distinct training stage and motivate rethinking how such data are used in language model training.