Homogenization of $\ell_2$-Adversarial Training in High-Dimensions: Exact Dynamics under Stochastic Gradient Descent
2026-06-30 • Machine Learning
Machine Learning
AI summaryⓘ
The authors create a mathematical tool to study how a type of machine learning called adversarial training progresses when data is Gaussian mixtures and the learning happens step-by-step in high dimensions. They turn complex random behaviors into simpler equations to understand the performance of different learning strategies, like Polyak stepsize and exact line search. Their work reveals why some standard approaches don't always improve learning steadily under adversarial conditions and identifies key factors causing this. They also develop a new equation that describes the learning process and connect the adversarial problem to a more classic regression problem with adjustable parameters.
adversarial trainingsingle-index modelsGaussian mixturesstochastic gradient descentPolyak stepsizeexact line searchanisotropic covarianceridge regressionstochastic differential equationhigh-dimensional limit
Authors
Fabrizzio Sabelli
Abstract
We develop a framework for analyzing the learning dynamics of $\ell_2$-adversarial training of single-index models on Gaussian mixtures in the high-dimensional limit under streaming stochastic gradient descent (SGD). We derive deterministic equivalents for a broad class of statistics of the SGD iterates, including the adversarial risk and distance to adversarial optimality, in terms of the solution to a system of ODEs. We use them to study two idealized learning rate schedules: the Polyak stepsize and exact line search. In the case of $\ell_2$-adversarial least squares with a single class, we show that, unlike noiseless standard least squares, no constant learning rate guarantees monotone descent of SGD towards a minimizer of the adversarial risk. We identify anisotropic covariance and a mismatch in ridge parameters as the main sources of suboptimality of exact line search relative to the Polyak stepsize. We also introduce a stochastic differential equation (SDE), called adversarial homogenized SGD, that captures the evolution of statistics of the iterates of SGD. For $\ell_2$-adversarial least squares, using this SDE, we show the evolution of the risk is equivalent, up to dimension-free constants, to that of SGD on standard least squares with an adaptive learning rate and adaptive $\ell_2$-regularization. When the dynamics converge, the limiting adversarial risk and SGD iterate are determined by a fixed-point equation, with the limiting iterate being equivalent to the solution of a ridge regression problem whose regularization parameter is the limiting effective regularization of SGD.