Improving accuracy of random neural network models with para-Gaussian methods
Para-Gaussian Approximations of Random Deep Neural Networks
Machine Learning
Summary
Randomly initialized neural networks are often approximated using Gaussian models, but this can become inaccurate for networks with a limited number of neurons or many layers. The authors propose a new method called the para-Gaussian approximation, which adds a small correction to better capture the network’s behavior. This new approximation uses mathematical tools called Hermite polynomials and allows for recursive calculations that improve predictions about deeper layers. They tested their approach on simpler linear networks and on networks using ReLU activation functions, showing how it can be applied in practice.
What this means in practice
- •For deep learning engineers: Estimate intermediate layer outputs more accurately in randomly initialized neural networks to guide analysis and design.
- •For signal processing teams: Improve modeling of deep signal transforms using para-Gaussian methods to better reflect non-Gaussian effects in layered systems.
Tested on simulated data.
Authors
Mihai Nica, Janosch Ortmann
Abstract
It is known that randomly initialized neural networks have Gaussian behaviour in the $n\to \infty$ limit. Unfortunately, this approximation can be poor for finite $n$ and gets worse as one goes deeper into the network. In this work, we develop a more accurate approximation that is \emph{slightly} non-Gaussian, which we call the para-Gaussian distribution and is constructed from the Hermite polynomials. We show how this para-Gaussian approximation can lead to a practical recursive method to approximate the deeper layers of the network. Finally, we apply this theory to linear neural networks (where things can be solved exactly) and to ReLU neural networks (where things can be approximated).