Dropout and Random Gradient Masking Are Asymptotically Equivalent in Large ResNets
Machine Learning
Summary
The authors compare two ways of adding randomness during deep learning training: dropout, which randomly hides parts of the network’s activations, and Random Gradient Masking (RaM), which randomly hides parts of the gradient updates instead. They find that for very large and deep ResNet models, both methods end up behaving the same way in the long run, even though they add randomness in different places. This means some common reasons given for why dropout works don’t explain RaM, but both still lead to similar learning behavior when the network is big enough. They also show this similarity extends to several variations of both methods.
Authors
Javier Maass, Lénaïc Chizat
Abstract
Dropout and Random Gradient Masking (RaM) are two training techniques used to improve performance in deep learning. Both techniques inject randomness into the training dynamics, but in significantly different ways: dropout applies random masks to the activations in the forward pass, whereas RaM leaves the forward pass unchanged and instead masks the gradients. In particular, the noise induced by RaM in the parameter updates is unbiased, so standard explanations for the effectiveness of dropout, such as the penalization effect or the prevention of co-adaptation between neurons, do not apply to RaM. In this work, we show that the difference between the two methods disappears for ResNets in the large depth and width asymptotics: in the complete feature learning regime, they both converge to the same large-scale limiting dynamics. This asymptotic equivalence holds for several variants of dropout and RaM, including layerwise dropout as used in stochastic-depth ResNets, albeit at slower quantitative rates. In fact, we also show that several of these variants collapse to the same limit asymptotically.