How Width and Data Shape Generalization Scaling Laws in Quadratic Neural Networks
2026-06-26 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors study how well a simple two-layer neural network learns depending on both its size and the amount of training data. They look at a specific setup where the network is trained with a certain regularization method and data has a special structure, allowing detailed analysis. They find different phases where the network's error scales in distinct ways based on the relationship between model size and data, influenced by properties of the target function. They also describe when the model starts to perfectly fit the training data and how this affects learning.
generalization errorscaling lawstwo-layer neural networkregularizationfinite-sample analysisfeature learningmodel widthempirical risk minimizationinterpolationspectral structure
Authors
Julius Girardin, Emanuele Troiani, Yizhou Xu, Vittorio Erba, Florent Krzakala, Lenka Zdeborová
Abstract
Understanding how performance scales jointly with model size and data is a central problem in modern machine learning. Existing theoretical works on scaling laws typically describe generalization as a function of data or compute, often in fixed-feature or infinite-width regimes and for online SGD. Here, we instead study how generalization scales with the number of trainable parameters and the number of samples in a feature-learning model. We analyze $\ell_2$-regularized empirical test error minimization in a quadratic two-layer network in a finite-sample setting with structured data. This setting allows for an explicit characterization of the generalization error as a function of the number of samples, model width, and regularization. Our results reveal a phase diagram with distinct scaling regimes as the number of parameters varies. In particular, the generalization error follows data-dependent power laws controlled by the spectral structure of the target. We further characterize the transitions between regimes, including the onset of interpolation, and their impact on generalization.