Adam optimization convergence slows with heavy noise and benefits from larger batches
On the Convergence of Vanilla Adam under Heavy-Tailed Noise and Index Sampling
Machine Learning
Summary
Training neural networks with the Adam optimizer can be tricky when the updates have heavy-tailed noise, which means the randomness in training is extreme and unpredictable. The authors studied how this kind of noise affects Adam's ability to find a good solution over time. They showed that when noise is more extreme, Adam's progress slows down and can depend heavily on how data samples are chosen. They also found that using bigger batches of data during training helps reduce this noisy effect and improves training stability.
What this means in practice
- •For machine learning engineers: Adjust batch sizes and learning rate schedules in training pipelines to improve convergence under noisy data conditions with heavy-tailed gradient noise.
- •For data-intensive system designers: Design sampling and data batching strategies that mitigate heavy-tailed noise effects on optimization convergence in large-scale machine learning systems.
Authors
Kanata Oowada, Hideaki Iiduka
Abstract
Understanding how stochastic gradient noise affects the convergence of Adam is an important problem in stochastic optimization, particularly when the noise exhibits heavy-tailed behavior. In this paper, we investigate the convergence of vanilla Adam in both the expected risk and the empirical risk minimization, focusing on the effects of heavy-tailed noise and its relationship to index sampling. For the expected risk minimization, we establish convergence guarantees under Hölder smoothness and bounded $\mathfrak{p}$-th moment assumptions, allowing stochastic gradient noise with infinite variance. Our analysis provides explicit upper bounds on the expected squared gradient norm and shows that the iteration-dependent convergence term decays more slowly as $\mathfrak{p}$ decreases. For the empirical risk minimization, we establish asymptotic and non-asymptotic convergence guarantees and demonstrate that heavy-tailed population noise can deteriorate convergence bounds through dataset-dependent constants as the dataset size increases. Furthermore, under Lipschitz smoothness, we formulate multistage Adam with practical learning rate schedules and demonstrate that increasing batch sizes can reduce index-sampling noise and improve convergence guarantees and SFO complexity. Numerical experiments support our theoretical findings on the effects of heavy-tailed noise and the computational benefits of increasing batch sizes.