Training multiclass models follows predictable stepwise learning patterns
SGD in Multiclass Logistic Regression: Sequential Learning and Scaling Laws
Machine Learning
Summary
Training models to recognize many different groups, like animals in photos, happens in steps where the model learns the most common groups first, then the rarer ones later. The authors studied how the training risk (or error) changes over time, finding it follows distinct phases and power laws especially when class frequencies drop off like a power law. They also figured out the best way to use limited computing resources by balancing model size and training time to reduce errors efficiently. These insights help explain how simple models behave when learning multiple categories, linking older linear models to modern neural network behavior.
What this means in practice
- •For machine learning engineers: Set model size and training duration optimally depending on available compute for large multiclass classification tasks.
- •For data platform architects: Design resource allocation strategies for multiclass model training that account for sequential class learning and compute constraints.
Authors
Konstantinos Christopher Tsiolis, Denny Wu, Christos Thrampoulidis, Murat A. Erdogdu
Abstract
We study the training dynamics of multiclass logistic regression on high-dimensional Gaussian mixture models with a large number of classes and establish precise scaling laws governing the cross-entropy risk under gradient-based optimization. We show that learning proceeds sequentially across classes, from most to least frequent. When the class priors follow a power law distribution, the risk dynamics decompose into three phases: an initial plateau until the first class is learned, a power-law decay regime during which sequential learning occurs, and a final convergence regime. We then analyze how model capacity interacts with optimization under a fixed compute budget. When the effective dimension is restricted via projection onto leading principal components, the risk decomposes into a capacity term (a power law in the retained dimension) and an optimization term (a power law in training time). Optimizing this tradeoff yields a compute-optimal scaling law for logistic regression, with explicit prescriptions for model size and training time as functions of compute. These results extend theoretical scaling laws from linear regression to multiclass classification, while connecting to empirical scaling laws observed in large-scale neural networks.