Deep network training converges better with noise and wider layers

Global Convergence in Deep Networks via the Second Law of Thermodynamics

Machine Learning

Summary

Training very large deep neural networks usually assumes the parameters don’t move far from where they start, but in reality, noise during training helps improve performance. The authors studied a training process that adds controlled noise to the network parameters and found that this noise helps keep key features stable even as the parameters change significantly. This stability allows the network to learn better without being stuck close to its starting state. Their math shows how wide the network needs to be and how much noise to add for training to reliably improve.

What this means in practice

  • For deep learning engineers: Design training protocols that use noise and network width to improve convergence guarantees without restricting parameter updates.
  • For optimization algorithm developers: Develop optimization methods leveraging stochastic noise to maintain kernel stability for better deep network training dynamics.

A theory result. No direct application yet.

Authors

Matan Tsipory, Ofir Gaash, Itamar Harel, Nathan Srebro, Daniel Soudry

Abstract

Existing convergence analyses of finite-width overparameterized deep neural networks under standard parameterization often rely on parameters remaining close to their initialization and typically neglect the role of noise, despite empirical evidence that stochasticity can improve optimization and generalization. To bridge this gap, we study $L_2$-regularized continuous-time Langevin dynamics (CLD) with isotropic noise, for training feedforward networks with smooth activations. We introduce a change-of-measure inequality, inspired by the second law of thermodynamics, that enables control of key properties of the training dynamics under the time-evolving parameter distribution. Leveraging a sharpened lower bound on the minimum eigenvalue of the Neural Tangent Kernel (NTK) at initialization, we show that injected noise helps preserve NTK stability with high probability without confining the weights to a small neighborhood of their initialization. As a result, we derive a non-asymptotic upper bound on the expected training loss. Our analysis yields explicit conditions on width and noise temperature that guarantee convergence to a finite loss floor. Finally, we experimentally show the network Jacobian can change from initialization, departing the commonly analyzed lazy regime.