Pretraining checkpoint quality affects later model training success
Good Pretraining, Bad SFT: Checkpoint Quality Across the Training Stack
Artificial IntelligenceComputation and Language
Summary
Many people assume that the best early version of a language model is the best starting point for further training. This paper shows that this is not always true in a complex 30-billion parameter model training setup. The researchers found that checkpoints that keep good performance even when slightly changed end up being better for later training steps. This means how stable a model’s solution is can predict how well it will do after more training.
What this means in practice
- •For machine learning engineers: Select pretraining checkpoints with higher solution density to improve fine-tuning results for large language models.
- •For ai service operators: Reduce model retraining time by using more robust pretraining checkpoints that maintain performance under small parameter changes.
Authors
Sohir Maskey, Philipp Scholl, Jonas Knupp, Pit Neitemeier, Sascha Wirges
Abstract
Language-model checkpoints are commonly selected by pretraining loss or benchmark scores, assuming that the highest-scoring checkpoint will remain the best starting point for subsequent training. We show that this assumption can fail in a full 30B mixture-of-experts training pipeline. The checkpoints that perform better after the full downstream training stack also have higher solution density, i.e., retain downstream performance under local weight perturbations.