AI summaryⓘ
The authors studied a mid-sized transformer model that is small enough to analyze completely to better understand grokking, which is when a model suddenly starts to generalize well after initially just memorizing the training data. They found that grokking depends heavily on how much of the training data the model sees and is very sensitive to small changes like numerical precision or hardware differences. Their results challenge previous single-run studies by showing grokking is a fragile phase transition best measured across multiple training runs with controlled conditions. They also discovered that breaking the task into smaller parts helps mostly because it reduces the training coverage needed, not because it provides extra hints.
GrokkingTransformerGeneralizationTraining-set coverageWeight decayPhase transitionModular arithmeticMulti-seed evaluationNumerical precisionModel interpretability
Abstract
Grokking--the delayed onset of generalization long after a network has fit its training set--is usually studied in models too large to read completely and reported from single training runs. We instead study a publicly released ~11,856-parameter Llama-style transformer (Glimmer-1-Base) on modular arithmetic, small enough to enumerate its weights, attention, and full input--output map, and we measure grokking as a multi-seed rate rather than a single outcome. In this fully-tractable regime grokking is a conditional, fragile phase transition. It is gated by training-set coverage, whose threshold tracks output cardinality (the modulus) more than task structure, an ordering that holds above the transition and across a ten-fold change in domain size. Weight decay reproduces the Omnigrok inverted-U at 12K parameters, a positive control on the rate measurement. Grokking also sits on a numerical knife-edge: two perturbations of the floating-point environment--CPU thread count (reduction order) and CPU-versus-GPU execution--each flip a minority of same-seed outcomes without a detectable shift in the aggregate rate. Decomposition into sub-task specialists helps chiefly by making coverage cheap rather than by adding supervision. Methodologically, multi-seed control under a fixed numerical environment overturns three dramatic single-run narratives in our own data, each a seed confound. The unit of evidence for grokking must therefore be a multi-seed rate under a pinned numerical environment, checked where possible against a direct reading of the model.