Symmetry properties explain infinite neural network training dynamics
The Symmetry of Differential Neural Tangent Kernel under the Orthogonal Group
Machine Learning
Summary
Training very large neural networks can be better understood using a mathematical tool called the differential neural tangent kernel (DNTK). This work explores how the DNTK behaves when you apply certain rotations (orthogonal transformations) to the input data and neural network parameters. The authors show that this kernel has symmetrical properties that simplify its structure and reveal how related mathematical operators share key characteristics. These insights help clarify the behavior of very wide neural networks, especially those used in physics-based applications.
What this means in practice
- •For machine learning engineers: Design neural networks with predictable training behaviors by leveraging DNTK symmetry under input rotations to simplify model analysis.
- •For computational physicists: Use insights on kernel symmetry to improve physics-informed neural networks for solving differential equations and modeling physical systems.
A theory result. No direct application yet.
Authors
Bangti Jin, Longjun Wu
Abstract
The differential neural tangent kernel (DNTK) is a tool for analyzing the training dynamics of overparameterized physics informed neural networks. In this work, we investigate the symmetry of the infinite DNTK under an orthogonal group, for two layer and deep neural networks separately, based on the explicit expression of the DNTK and recursive formula, respectively. The results are for all activations with or without a bias term, and shed insight into the structure of the infinite width DNTK. Then we prove that the integral operators over the unit sphere induced by operators from the same group orbit share the spectrum, and their reproducing kernel Hilbert spaces differ only by an orthogonal transformation, and that the operator fixed DNTK exhibits zonal symmetry on the unit sphere for symmetric operators. Thus the eigenvalue decay rate of the associated integral operator can be derived for shallow networks with RePU activation and zero bias initialization.