Function space analysis reveals stable learning patterns in neural networks
A Function-Space Approach to the Statistical Mechanics of Learning Dynamics
Artificial Intelligence
Summary
Training deep neural networks is complex because they have many parameters changing in complicated ways. The authors study the learning process by focusing on the functions these networks compute, rather than individual parameters. They find a mathematical way to describe how errors shrink during learning, showing that networks tend to learn faster along smooth, structured directions in the function space. This approach offers a new way to understand how neural networks organize learning despite their complexity.
What this means in practice
- •For machine learning engineers: Improve training algorithms by focusing on function space directions that guarantee faster error reduction and stable learning.
- •For signal processing teams: Use function-space stability insights to design adaptive filters that relax faster along smooth structural features in data.
A theory result. No direct application yet.
Authors
Yizhou Zhang, Weichen Wu, Lun Du, Zhengjie Miao
Abstract
Deep neural networks exhibit regular macroscopic behavior despite highly nonlinear dynamics in vast parameter spaces. We develop a statistical-mechanical description of learning directly in function space, treating parameter configurations as microscopic realizations and functions with their dynamical operators as macroscopic variables. For mean-squared loss, the exact error dynamics are governed by the learning operator \(M=JJ^\ast\). Combining the dynamical Boltzmann weight of the conditional stochastic dynamics with the parameter-space density of states, whose local curvature defines a statistical operator \(B\), and integrating over local fluctuations yields $$ Φ_{\mathrm{fluc}}(M;B)=\frac{σ_ξ^2}{2}\log\det(M^{-1}+B)+\mathrm{const}. $$ At fixed spectrum, this term is rotationally stationary when \([M,B]=0\), is minimized by pairing large eigenvalues of \(M\) with small eigenvalues of \(B\), and generates a local restoring contribution against rotational mismatch. For ReLU-type function spaces under mild stable statistical conditions, \(B=σ_ξ^2L^\ast\mathcal K L\), where \(L\) measures coarse-grained second-order structure. Thus the low-\(B\) sector corresponds, up to bounded anisotropy of \(\mathcal K\), to low structural curvature, implying a preference for faster relaxation along smooth, data-adaptive directions. These results identify function space as a natural macroscopic level for studying stable collective organization in learning.