Summary
Training models to recognize many different categories can be tricky, especially when some categories are much more common than others. The researchers found that when training a type of model called multiclass logistic regression, the model learns categories one by one, starting with the most common. They also discovered patterns in how the model improves over time and how the number of model features and training time interact under limited computing resources. Their work helps explain how to balance model size and training time to get the best results with given computing power. This builds on earlier findings for simpler models and relates to behavior seen in big neural networks.
multiclass logistic regressiongradient descentGaussian mixture modelscross-entropy riskpower law distributiontraining dynamicsmodel capacityprincipal component analysisscaling lawscompute budget
Authors
Konstantinos Christopher Tsiolis, Denny Wu, Christos Thrampoulidis, Murat A. Erdogdu
Abstract
We study the training dynamics of multiclass logistic regression on high-dimensional Gaussian mixture models with a large number of classes and establish precise scaling laws governing the cross-entropy risk under gradient-based optimization. We show that learning proceeds sequentially across classes, from most to least frequent. When the class priors follow a power law distribution, the risk dynamics decompose into three phases: an initial plateau until the first class is learned, a power-law decay regime during which sequential learning occurs, and a final convergence regime. We then analyze how model capacity interacts with optimization under a fixed compute budget. When the effective dimension is restricted via projection onto leading principal components, the risk decomposes into a capacity term (a power law in the retained dimension) and an optimization term (a power law in training time). Optimizing this tradeoff yields a compute-optimal scaling law for logistic regression, with explicit prescriptions for model size and training time as functions of compute. These results extend theoretical scaling laws from linear regression to multiclass classification, while connecting to empirical scaling laws observed in large-scale neural networks.