Large models resist early training biases better than small models
Sequential Pretraining Favors Large Models
Machine Learning
Summary
When training AI models on different types of data one after another, small models tend to focus too much on the earlier data and struggle to learn from later data that may be more important. The authors show that bigger models handle this problem better because they can allocate their learning more efficiently. They introduce a method called Exposure Therapy that helps smaller models learn from later data better by reducing this early-data bias. Their experiments show that this method improves the overall ability of smaller models to learn complex tasks.
What this means in practice
- •For machine learning engineers: Improve smaller AI models’ learning from sequential training data by reducing bias toward early data distributions.
- •For ai model trainers: Increase effectiveness of foundation model pretraining on complex tasks like code and reasoning without needing very large models.
Authors
Mohnish Harwani, Yujia Zheng
Abstract
Large neural networks often acquire capabilities that small models fail to learn. Does this stem from large models learning more representative features, or from being more robust to unaccounted-for adverse effects introduced during training? We define and quantify one such adverse effect, primacy bias, as the extent to which exposure to early data distributions impairs later learning. We show that small models can allocate learning capacity inefficiently toward early distributions, whereas sufficiently overparameterized models are robust to this effect. This inefficiency is particularly consequential in pretraining, where foundation models often encounter heterogeneous data distributions sequentially rather than jointly. As a result, small foundation models can struggle to learn distributions encountered late in training, which is particularly harmful when later data emphasizes desirable capabilities such as code, mathematics, and reasoning. Motivated by these findings, we introduce Exposure Therapy (ET), a simple regularization that promotes more efficient allocation of learning capacity during sequential pretraining. We demonstrate that ET improves foundation models' performance on late data distributions as well as overall capability in models up to the billion-parameter scale. Overall, our results suggest that some benefits of large foundation models may arise from greater robustness to adverse training effects, rather than from learning more representative features, and that improved training algorithms can recover some of these advantages in smaller models.