Memorizing networks differ from generalizing networks in test behavior and training costs
Memorization in image classification: test set behavior and scaling laws
Machine Learning
Summary
Deep learning networks can either learn useful patterns or just memorize random data. This paper looks at how networks that memorize behave differently on new images compared to those that learn general patterns. The authors find that memorizing networks trained from different starting points make different predictions on test images, unlike generalizing networks. They also discover that as the dataset grows, the amount of training needed to memorize all examples can actually go down, even if the data has no meaningful patterns.
What this means in practice
- •For machine learning engineers: Design training regimes by understanding how dataset size affects memorization time and model capacity requirements for image tasks with noisy labels.
- •For ai system auditors: Identify whether deep models memorize or generalize by comparing predictions of independently trained models on given test sets.
Authors
Jeevesh Attri, Florentin Guth
Abstract
Deep neural networks are capable of both learning generalizing solutions on natural tasks and memorizing arbitrary mappings. We argue that understanding the former requires understanding the latter, which is much less studied. In what ways do networks that generalize or memorize differ in their behavior or their training dynamics? Do memorizing networks simply function as a lookup table where input-output mappings are stored independently? We investigate these questions in the context of CNNs trained on image classification tasks with randomized labels or pixels. We first study the behavior of such memorizing networks on unseen images. We find that independently initialized networks trained on the same random labels learn different predictive functions: they make different predictions on test examples, in contrast with generalizing networks. We then examine the effect of dataset size on memorization through the required network capacity and training time to interpolate the training set. Counter-intuitively, we find that the number of presentations of a given example required for complete memorization can decrease with dataset size even when input-output pairs share no common structure. Although memorizing networks do not learn a universal solution, they exhibit interesting behaviors which call for further investigations into the underlying mechanisms.