What Drives the Inlier-Memorization Effect? A Theory of Outlier Detection via Early Training Dynamics
2026-06-29 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors study why deep learning models tend to learn normal data patterns faster than unusual ones, a behavior called the inlier-memorization (IM) effect, which helps spot outliers. They focus on a simple autoencoder model and prove that under certain mild conditions, it remembers normal data but not outliers early in training. They also explain how this effect starts, how strong it is, and how long it lasts, depending on data and model setup. Using these insights, the authors suggest practical ways to improve outlier detection and achieve better results on benchmark datasets.
outlier detectioninlier-memorization effectautoencoderunsupervised learninganomaly detectionmodel initializationdata preprocessingearly training behaviorsADBencH datasets
Authors
Kunwoong Kim, Dongha Kim
Abstract
Outlier detection (OD) aims to identify anomalous instances by learning the underlying structure of normal data (inliers), and is particularly challenging in fully unsupervised settings where no information about anomalies is available during training. Recent advances have leveraged the inlier-memorization (IM) effect, a phenomenon in which deep models memorize inlier patterns earlier than those of outliers, as a powerful signal for distinguishing outliers. However, despite its empirical success, the theoretical understanding of the IM effect remains limited. In this work, we present a theoretical study of the IM effect. Focusing on a simple autoencoder, we show that, under mild assumptions, the model can successfully memorize inliers while failing to memorize outliers during certain stages of early training. In particular, we characterize not only the emergence of the IM effect, but also its strength and persistence, and analyze how these properties depend on the data distribution and parameter initialization. In addition, building on these insights, we derive simple yet practical guidelines for enhancing the IM effect, including data preprocessing and parameter initialization schemes, achieving state-of-the-art performance on the ADBench datasets. Our findings provide a theoretical foundation for the IM effect and offer actionable directions for improving IM-based outlier detection methods.