Training dynamics of deep ResNets simplify to Gaussian mean ODEs

Infinite ResNets are Gaussian Mean ODEs

Machine Learning

Summary

Understanding how very deep neural networks learn is difficult because of complex behaviors. The authors show that when ResNet models get infinitely large in width and depth, their training can be described by a simpler mathematical system called a Gaussian Mean ODE. This means the complicated process of training can be seen as a gradient descent on this simpler system, making analysis and perhaps design easier. They also extend this approach to models with multiple experts, showing similar simplifications hold.

What this means in practice

  • For deep learning engineers: Predict training dynamics of very deep ResNet architectures using a simplified ODE model to assist in network design and debugging.
  • For machine learning platform developers: Incorporate large-scale limit analysis into training simulations to optimize resource allocation when deploying wide and deep neural networks.

A theory result. No direct application yet.

Authors

Louis-Pierre Chaintron, Lénaïc Chizat, Javier Maass

Abstract

Understanding the training dynamics of deep neural networks in large-scale limits is a question both of foundational and of practical importance. Prior works have obtained such descriptions through dynamical mean field theory (DMFT), yielding a system of non-Markovian dynamics that are challenging to analyze and that obscure the optimization structure of the original dynamics. In this paper, we obtain instead a simple description of the large-scale limit training dynamics for ResNets in standard shape scalings and in the $1/\text{depth}$ branch scaling that allows maximal (local) feature updates. For a model trained with gradient descent (GD) from a random initialization, the limit dynamics is simply GD on a Neural Mean ODE with Gaussian initial weights. Our analysis covers population or empirical risk GD in the joint limit of large width, depth and embedding, input, and output dimensions. We obtain approximation scaling laws that depend on the shape of the ResNet and the covariance spectrum of the data. We also illustrate the flexibility of our approach by discussing Mixture of Expert models, where likewise the limit admits an intuitive (nested) Mean ODE description.