Language models switch suddenly between memorizing and reasoning during training
Generalization Dynamics of LM Pre-training
Computation and Language
Summary
People often think language models steadily improve from just repeating patterns to genuinely understanding and reasoning during their training. This paper shows that instead, these models flip back and forth between simple memorization and deeper reasoning many times as they learn. These switches can’t be fixed by standard training tricks and happen because the model’s limited capacity forces it to choose between shallow and deep strategies. The authors provide a small test set to study this behavior and show how to pick training stages and data that better capture reasoning abilities.
What this means in practice
- •For machine learning engineers: Find checkpoints during language model training that generalize reasoning better than final or intermediate versions to improve downstream model performance.
- •For data curation teams: Choose and arrange pre-training data to reduce instability in language model generalization during training and improve model reliability.
Authors
Jiaxin Wen, Zhengxuan Wu, Dawn Song, Lijie Chen
Abstract
People typically assume that LMs stably mature from pattern-matching parrots to generalizable intelligence during pre-training. We build a toy eval suite and show this mental model is wrong: throughout pre-training, LMs frequently and suddenly hop between parrot-like and intelligence-like computations. We call this mode-hopping. Across our suite, LMs suddenly latch onto memorized or in-context patterns instead of in-context learning, use System 1 instead of System 2 thinking, pick up what sounds true instead of what is true, fail at multi-hop persona QA, out-of-context reasoning, and emergent misalignment -- then just as suddenly revert and generalize. Mode-hopping is not explained by standard optimization dynamics: it is locally stable and cannot be fixed by checkpoint averaging. We instead think of it as a capacity allocation problem: in a capacity-bounded model, generalizable circuits must compete with the shallow ones learned early in training, and the data in each pre-training window may decide which circuits win. Our suite provides a new efficient lens on generalization. We demonstrate two concrete applications: (i) select intermediate pre-training checkpoints that strongly generalize reasoning and alignment, better than the final pre- or mid-training checkpoints, and (ii) select pre-training data that controls and stabilizes generalization dynamics.