What do Language Models Learn and When? The Implicit Curriculum Hypothesis
2026-04-09 • Computation and Language
Computation and Language
AI summaryⓘ
The authors studied how large language models (LLMs) learn different abilities during their training. They found that LLMs acquire skills in a consistent and predictable order, where simpler skills emerge before more complex ones that build on them. By analyzing model behavior on a set of simple tasks, they showed this learning order is visible in the internal workings of the models and can help predict when new composite skills will appear. This means training is more organized than just looking at overall performance improvements would suggest.
large language modelspretrainingemergencescaling lawscompositional tasksmodel representationslogical reasoningcoreferencemorphological transformationstraining trajectory
Authors
Emmy Liu, Kaiser Sun, Millicent Li, Isabelle Lee, Lindia Tjuatja, Jen-tse Huang, Graham Neubig
Abstract
Large language models (LLMs) can perform remarkably complex tasks, yet the fine-grained details of how these capabilities emerge during pretraining remain poorly understood. Scaling laws on validation loss tell us how much a model improves with additional compute, but not what skills it acquires in which order. To remedy this, we propose the Implicit Curriculum Hypothesis: pretraining follows a compositional and predictable curriculum across models and data mixtures. We test this by designing a suite of simple, composable tasks spanning retrieval, morphological transformations, coreference, logical reasoning, and mathematics. Using these tasks, we track emergence points across four model families spanning sizes from 410M-13B parameters. We find that emergence orderings of when models reach fixed accuracy thresholds are strikingly consistent ($ρ= .81$ across 45 model pairs), and that composite tasks most often emerge after their component tasks. Furthermore, we find that this structure is encoded in model representations: tasks with similar function vector representations also tend to follow similar trajectories in training. By using the space of representations derived from our task set, we can effectively predict the training trajectories of simple held-out compositional tasks throughout the course of pretraining ($R^2 = .68$-$.84$ across models) without previously evaluating them. Together, these results suggest that pretraining is more structured than loss curves reveal: skills emerge in a compositional order that is consistent across models and readable from their internals.