Theory improves early stopping to cut deep learning training costs
Theory-Guided Instance-Dependent Early Stopping via Second-Order Loss Dynamics
Machine LearningComputer Vision and Pattern Recognition
Summary
Training deep learning models can be slow because every example needs full processing multiple times. This paper studies a method that skips extra processing for examples whose difficulty no longer changes much, saving time. The authors provide a mathematical analysis that explains when and why a particular way of measuring change in loss works well, and how to set thresholds automatically. Their method saves around half the computation on popular image datasets while keeping accuracy nearly the same. It also works on a smaller vision transformer model, though with smaller savings.
What this means in practice
- •For machine learning engineers: Reduce training time on deep vision models by skipping updates for stable examples using theory-based thresholds.
- •For edge ai developers: Lower computation costs when training resource-constrained models like vision transformers by applying calibrated early stopping on instances.
Authors
Ahad Bin Islam Shoeb, Jamal Uddin Tanvin, Kamrul Hasan, Alve Reduan, Imtiaz Ahmed
Abstract
Instance-dependent early stopping (IES) reduces training computation by skipping backpropagation for examples whose losses have stabilized, but the second-order loss-difference criterion used by IES lacks a theoretical justification. We develop Theory-Guided Instance-Dependent Early Stopping (TG-IES), which analyzes this criterion under explicit assumptions and derives a noise-calibrated mastery threshold and a conditional stability bound for instance exclusion. We show that second-order differencing has a favorable bias-variance trade-off over first- and third-order differences in a specific signal-to-noise regime, and that its signal contains both gradient-drift and curvature terms. TG-IES derives its stopping threshold and patience from training-run statistics rather than using the manually selected threshold of IES. Across CIFAR-10 and CIFAR-100, TG-IES saves 53-58% of backpropagation instances in its main convolutional settings while keeping accuracy within 0.1-0.9 percentage points of full training; in the primary ResNet-18/CIFAR-10 setting, this corresponds to 35.7% FLOP-normalized savings. The method also transfers to ViT-Tiny, where it achieves smaller savings and incurs a modest accuracy cost. These results provide a theoretical basis for second-order instance-dependent early stopping and show that theory-guided calibration can substantially reduce training computation under the assumptions and experimental settings studied.