Mean field fluctuations reveal uncertainty in neural network training outputs

Fluctuations of Nonlinear Observables in Mean Field Neural Network Training

Machine Learning

Summary

Training wide neural networks involves many parameters that change over time, and scientists use mathematical models to understand this process in large networks. This paper looks at how small random variations in these parameters affect the specific quantities people actually care about, like predictions or measures derived from the network. The authors provide a way to calculate how these fluctuations influence these quantities and determine when the observations fully capture the information needed. This helps quantify uncertainty when working with networks of finite size and understand if the chosen data reveals everything important.

What this means in practice

  • For machine learning engineers: Quantify sources of uncertainty in neural network outputs when training with realistic finite-width models.
  • For statistical modelers: Assess whether chosen measurements from a neural network fully reveal targeted quantities of interest using a differential condition.

A theory result. No direct application yet.

Authors

Arnaud Descours, Geoffrey Lacour

Abstract

Mean field limits describe the training dynamics of wide neural networks through the evolution of the empirical distribution of their parameters. Although functional central limit theorems characterize the asymptotic fluctuations of this distribution, quantities of practical interest are typically nonlinear observables of the parameter distribution rather than the distribution itself. In this work, we show how these mean field fluctuations propagate to finite dimensional nonlinear observables for shallow neural networks trained by stochastic gradient descent. Working in the weighted Sobolev space in which the limiting fluctuation process is constructed, we apply a functional Delta method under ordinary Fr{é}chet differentiability, without requiring Lions derivatives with respect to the measure variable. We obtain a central limit theorem for the observables and, under a suitable representation of their differentials, an explicit covariance formula inherited from the underlying mean field fluctuation theory. We also study whether prescribed quantities of interest can be recovered from the selected observations. Under a constant rank assumption, we prove that a quantity of interest factors locally through the observation functional if and only if, throughout a neighborhood, the kernel of the differential of the observation is contained in that of the quantity of interest. Thus, a differential condition expressed directly in the ambient Sobolev space yields an exact nonlinear local factorization. These results provide a framework both for quantifying finite-width uncertainty on observable, statistically or physically meaningful quantities and for assessing whether the chosen observations contain the information required to identify them.