Papers for

neural network researchers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Neural network learning examined through spectral renormalization group theory

A Spectral Renormalization-Group View of Learning

Abstract: We study how the layers in deep neural networks (NNs) converge, using Wilsonian Renormalization-Group (RG) ideas to organize the spectral picture. During training, these layers often develop heavy-tailed power-law (PL) spectra, and the strongest checkpoints often appear near the PL exponent $α\approx 2$. This pattern raises a simple question: why should this value mark a well-trained layer? The answer is both statistical and physical. Our statistical hypothesis is that $α=2$ marks the boundary between stable if not optimal learning and overfitting via non-self-averaging. The physical answer is that the same value makes the spectral energy gain scale evenly across logarithmic spectral scales, which is the proposed Wilsonian marginal condition. We then apply the ERG trace-log condition, the proposed SETOL theory, which we justify by taking volume-preserving, scale-invariant transformations to remove arbitrary layer scale along the proposed RG flow. The resulting spectral RG flow connects the HTSR and SETOL theories to optimizers such as Muon, and motivates a new optimizer extension, WW-PGD. Together, these results point toward a spectral Renormalization-Group view of learning, in which NNs may approach shared spectral structure in the settings studied here.

Fri 30 OctMachine LearningArtificial Intelligence
The gist
Training layers of deep neural networks sometimes show patterns in their weight structures that follow a power-law distribution. The authors investigate why a particular power-law exponent around 2 often signals a well-trained layer, suggesting this marks a balance between stable learning and overfitting. They use ideas from physics called the Renormalization Group to explain how these patterns emerge and relate them to optimizer design. Their work proposes a new way to view learning as a flow of spectral features across layers, leading to more informed optimization methods.
Open → 2611.00422v1

Continuous Thought Machine traces image classifications along a fixed pattern

Riding the Fixed-Point Curve: Image Classification in the Continuous Thought Machine

Abstract: The Continuous Thought Machine (CTM) is a recurrent network that attends to inputs and makes predictions based on the synchronization between its neurons. On tasks spanning many domains, it displays emergent thought processes that have not been explained. We report a number of findings about how the CTM classifies images, focusing on ImageNet. Each thinking step is an attracting but strongly non-normal map exhibiting balanced amplification (Murphy and Miller, 2009); while attraction is already present at CTM initialization, training highly promotes non-normality. By Banach's theorem, the neural state has a locally unique attracting fixed point, which moves at every step, and the thought trajectory is the curve these fixed points trace, followed one tick behind. Most of that curve's motion is shared across images, and visible as a previously reported traveling wave of activity across neurons. The attention schedule along the thought is largely generic: shuffling queries across images at every tick costs only 7 percent of classification accuracy, while image-independent increments predict the next query with R^2 = 0.98 (held out). Our results suggest that the CTM has found a single strategy to process all ImageNet images.

Tue 13 OctMachine LearningComputer Vision and Pattern RecognitionNeural and Evolutionary Computing
The gist
Classifying images is a complex task, and this paper looks at how a special type of neural network called the Continuous Thought Machine (CTM) does it. The network processes images step-by-step by moving through a series of stable states, like following a path of fixed points that change over time. Most of this processing path is shared for all images, which means the CTM uses a common strategy to understand different pictures. The authors also found that swapping parts of the network’s attention does not harm accuracy much, showing this process is very stable and generic.
Open → 2610.19220v1