Implicit Regularization and Generalization in Overparameterized Neural Networks
2026-04-08 • Machine Learning
Machine Learning
AI summaryⓘ
The authors studied why very large neural networks don't just memorize training data but still perform well on new data, which classical theory struggles to explain. They ran experiments showing that smaller batch sizes during training lead to better generalization and flatter points in the loss landscape. They also found that smaller sparse networks, made by pruning most parameters, can almost match the full model's accuracy. Their work suggests that how the network architecture, training methods, and landscape geometry work together is key to understanding generalization in big models.
OverparameterizationGeneralizationStochastic Gradient DescentBatch SizeLoss LandscapeHessian EigenvaluesNeural Tangent KernelDouble DescentLottery Ticket HypothesisPruning
Authors
Zeran Johannsen
Abstract
Classical statistical learning theory predicts that overparameterized models should exhibit severe overfitting, yet modern deep neural networks with far more parameters than training samples consistently generalize well. This contradiction has become a central theoretical question in machine learning. This study investigates the role of optimization dynamics and implicit regularization in enabling generalization in overparameterized neural networks through controlled experiments. We examine stochastic gradient descent (SGD) across batch sizes, the geometry of flat versus sharp minima via Hessian eigenvalue estimation and weight perturbation analysis, the Neural Tangent Kernel (NTK) regime through wide-network experiments, double descent across model scales, and the Lottery Ticket Hypothesis through iterative magnitude pruning. All experiments use PyTorch on CIFAR-10 and MNIST with multiple random seeds. Our findings demonstrate that generalization is strongly influenced by the interaction between network architecture, optimization algorithms, and loss landscape geometry. Smaller batch sizes consistently produced lower test error and flatter minima, with an 11.8x difference in top Hessian eigenvalue between small-batch and large-batch solutions corresponding to 1.61 percentage points higher test accuracy. Sparse subnetworks retaining only 10% of parameters achieved within 1.15 percentage points of full model performance when retrained from their original initialization. These results highlight the need for revised learning-theoretic frameworks capable of explaining generalization in high-dimensional model regimes.