Summary
People usually pick AI models that do best on tests or training to improve them further. The researchers show that this approach can fail when working with very large AI models called mixture-of-experts. Some models that don't score highest initially actually turn out better after more training because they handle small changes well. The team found that models with what they call higher 'solution density' tend to keep their good performance even if their settings are tweaked a bit.
language modelcheckpointpretrainingdownstream trainingmixture-of-expertsloss functionsolution densityweight perturbationsbenchmark score
Authors
Sohir Maskey, Philipp Scholl, Jonas Knupp, Pit Neitemeier, Sascha Wirges
Abstract
Language-model checkpoints are commonly selected by pretraining loss or benchmark scores, assuming that the highest-scoring checkpoint will remain the best starting point for subsequent training. We show that this assumption can fail in a full 30B mixture-of-experts training pipeline. The checkpoints that perform better after the full downstream training stack also have higher solution density, i.e., retain downstream performance under local weight perturbations.