Deep learning accuracy can dip then recover during training
The Accuracy Well: Nonmonotone Accuracy Dynamics at Vanishing Initialization
Machine Learning
Summary
When training deep learning models starting from very small values, accuracy can go up quickly at first, then drop down for a while, and finally recover later on. The authors call this the accuracy well. They explain it happens across different types of networks and datasets. They also prove mathematically that accuracy can improve even when changes in model loss are very small, and sometimes accuracy can go from perfect to terrible and back. This shows that accuracy can give insights into learning that loss values alone cannot.
What this means in practice
- •For machine learning engineers: Detect accuracy dips during training to adjust hyperparameters or stopping criteria for better model performance.
- •For ai software developers: Develop diagnostic tools showing accuracy well behavior to better understand and visualize model training progress.
Authors
Yaroslav Gusev, Eugene Golikov, Dmitry Yarotsky
Abstract
We study classification dynamics under small initialization and identify a pronounced non-monotone behavior of accuracy that we call the accuracy well: accuracy rises rapidly early in training, subsequently drops, and only much later recovers. We observe this phenomenon in deep linear networks, fully connected ReLU networks on MNIST, deep convolutional networks on CIFAR-10, and controlled synthetic tasks, across different losses and optimizers and in both training and test accuracy. We propose a three-stage explanation in terms of target-driven organization, a degeneracy bottleneck, and subsequent recruitment and recovery. We develop a rigorous theory of these dynamics for shallow and deep linear networks, showing in particular that high accuracy can emerge while the predictor and the change in loss remain asymptotically vanishing, before continued optimization causes accuracy to deteriorate. For a simple synthetic classification task, we prove an extreme instance in which accuracy asymptotically rises to 100%, collapses to 0, and later recovers permanently to 100%. Our results show that classification accuracy can reveal aspects of learning dynamics that are not captured by the training loss alone.