Double descent explained as energy distribution in model training
Double descent is the principle of least action
Machine LearningArtificial Intelligence
Summary
When building computer models, sometimes making them bigger first makes mistakes worse, then better again—a pattern called double descent. The authors explain this by thinking of the training process like a particle moving on a landscape of hills and valleys, with temperature affecting its path. Adding more parts to the model spreads out the 'energy,' effectively helping to control complexity and reduce errors. This explanation connects ideas from physics to why bigger models can sometimes perform better.
What this means in practice
- •For machine learning engineers: Predict how increasing model size affects training dynamics and generalization at fixed loss values.
- •For statistical physicists working on optimization: Apply statistical mechanics concepts to analyze optimization behavior in high-dimensional parameter spaces.
A theory result. No direct application yet.
Authors
Congzhou M Sha
Abstract
The test error of a model plotted against its number of parameters $d$ falls, peaks when the model can just fit the training data, and falls again, exhibiting the double descent phenomenon. We explain the phenomenon with statistical mechanics. The training trajectory of a stochastic gradient-based method is a particle wandering over the energy landscape of the training loss at an induced temperature $T$, and a run that has equilibrated visits every parameter vector of a given training loss equally often, the fundamental postulate of statistical mechanics, with probability given by the Boltzmann distribution. Because training starts at an initial point and has only finite time to diffuse, it carries an effective weight decay, which makes every parameter a quadratic degree of freedom. The equipartition theorem then distributes the energy among the $d$ degrees of freedom in shares of $T/2$, so at a fixed training loss adding parameters lowers the temperature and drives the Boltzmann distribution toward the stationary path. Finally, adding parameters can only lower the $L^2$ norm of the stationary path, so a solution sampled at fixed loss is less likely to be large with increasing $d$, effectively increasing weight regularization.