Neural network learning examined through spectral renormalization group theory
A Spectral Renormalization-Group View of Learning
Machine LearningArtificial Intelligence
Summary
Training layers of deep neural networks sometimes show patterns in their weight structures that follow a power-law distribution. The authors investigate why a particular power-law exponent around 2 often signals a well-trained layer, suggesting this marks a balance between stable learning and overfitting. They use ideas from physics called the Renormalization Group to explain how these patterns emerge and relate them to optimizer design. Their work proposes a new way to view learning as a flow of spectral features across layers, leading to more informed optimization methods.
What this means in practice
- •For machine learning engineers: Design optimizers that control layer training dynamics by targeting spectral properties linked to stable learning.
- •For neural network researchers: Develop theoretical frameworks connecting spectral properties of layers to training behavior and generalization.
A theory result. No direct application yet.
Authors
Charles H. Martin
Abstract
We study how the layers in deep neural networks (NNs) converge, using Wilsonian Renormalization-Group (RG) ideas to organize the spectral picture. During training, these layers often develop heavy-tailed power-law (PL) spectra, and the strongest checkpoints often appear near the PL exponent $α\approx 2$. This pattern raises a simple question: why should this value mark a well-trained layer? The answer is both statistical and physical. Our statistical hypothesis is that $α=2$ marks the boundary between stable if not optimal learning and overfitting via non-self-averaging. The physical answer is that the same value makes the spectral energy gain scale evenly across logarithmic spectral scales, which is the proposed Wilsonian marginal condition. We then apply the ERG trace-log condition, the proposed SETOL theory, which we justify by taking volume-preserving, scale-invariant transformations to remove arbitrary layer scale along the proposed RG flow. The resulting spectral RG flow connects the HTSR and SETOL theories to optimizers such as Muon, and motivates a new optimizer extension, WW-PGD. Together, these results point toward a spectral Renormalization-Group view of learning, in which NNs may approach shared spectral structure in the settings studied here.