Neural network size must grow to find complex patterns early in learning
Representation Complexity Along Learning Trajectories: Trading Width for Precision and Time
Machine LearningArtificial Intelligence
Summary
This paper studies why big neural networks use more complicated internal steps when learning than needed for their final task. The authors show that finding complex patterns can require much larger network capacity than just representing them after learning. Using mathematical tools, they prove limits on how small these networks can be at the start before they can detect a signal through learning. Their experiments confirm these predictions and show how the tradeoff between network size, precision, and time affects learning.
What this means in practice
- •For neural network engineers: Estimate minimal network size needed early in training for complex pattern discovery to guide architecture choices.
- •For machine learning optimization teams: Inform the design of training schedules that balance model width and precision to improve learning efficiency.
A theory result. No direct application yet.
Authors
Ryan Singh, Miguel De Llanza, Christopher L. Buckley
Abstract
What computational role does overparameterisation play during learning? Information-plane analyses suggest that neural networks can traverse representations substantially more complex than those required by their final predictors. We explore when this transient complexity serves a computational purpose by comparing the representation complexity needed for prediction with that needed to make an unknown target statistically visible through gradient feedback. We introduce an oracle model of neural network learning and use Kolmogorov \(n\)-widths to quantify effective representation complexity. Our main technical result transfers complexity bounds from the forward representation to the corresponding gradient-query features. Combined with statistical-query lower bounds, this gives a task-agnostic lower bound on the initial effective width required for target-dependent gradient information to become detectable. For \(k\)-sparse parity in \(D\) dimensions, a known target admits an \(O(k)\)-width representation, while the general bound already implies a polynomial separation between representation and discovery. We then exploit parity's invariance under coordinatewise sign reflections to identify a sharper unsigned effective width, which removes sign distinctions invisible to the parity information channel. At a transfer-compatible measurement resolution, its critical scale is \(\widetildeΘ_k\left( D^{k-1}τ^2 \right)\), up to logarithmic factors. Thus, the representation resources required to discover a predictor can substantially exceed those required to represent it once learned. Interpreting precision through accumulation over fresh samples yields a corresponding width-time frontier. Experiments support the predicted dimension dependence and examine how optimisation and generalisation depart from this idealised frontier.