Neural network training simplified using smaller state spaces
Dynamical coarse-graining of neural-network training
Machine LearningNeural and Evolutionary Computing
Summary
Training neural networks typically involves handling an enormous number of parameters, which can be complex and slow. The authors present a way to simplify this by focusing on key summary features, or 'moments,' reducing the problem to a much smaller space that still accurately predicts learning behavior. They prove several theorems that support this approach and demonstrate it on a specific type of neural network. This method could help design simpler models that capture the essence of how training evolves.
What this means in practice
- •For machine learning engineers: Create reduced models to predict and control training without processing all neural network parameters directly.
- •For control systems designers: Design control strategies for neural network training processes using compact state-space models derived from key observables.
Tested on one dataset.
Authors
Lin-Juan Wang, Da Li, Huiling Duan, Jianxiang Wang
Abstract
Dynamical coarse-graining of neural networks gives macroscopic observables and enables training to be predicted, and controlled in a state space far smaller than the parameter space. In this paper, we first define the learning operator following the training law of neural networks and establish a descent theorem for the learning operator, which serves to determine whether the operator is sufficient to characterize the learning dynamics. Then the exact closure theorem and the dynamic coarse-graining theorem are presented in sequence. The three theorems established in this paper provide a systematic framework for constructing and assessing dynamical coarse-grained models of neural-network training. Finally, we demonstrate the applicability of the theorems with a finite-width nonlinear ReLU network. Its dynamic coarse-grained model for a fixed pattern is exactly derived and composed of five independent moments. The model exactly predict the training outputs and evolving learning operator. The framework may also guide the development of approximate coarse-grained models for more general neural networks when exact closure is unavailable.