Logistic gradient descent paths clarified for large initialization scales
Ideal Paths for Approximating Logistic Gradient Descent Trajectories at Large Initialization
Machine Learning
Summary
Training machine learning models often starts from an already trained model, but it was unclear how this starting point affects the training’s progress. The authors studied how logistic gradient descent behaves when the starting model is large in scale, breaking down the training path into two clear stages. They created an idealized path that closely matches real training trajectories at large scales and showed how the loss behaves along this path. Their results help explain intermediate training dynamics that previous theories did not address.
What this means in practice
- •For machine learning engineers: Predict intermediate behavior of training large models initialized from prior ones to better monitor and adjust training.
- •For applied mathematicians: Use the geometric approximation to analyze and design gradient-based algorithms on linearly separable data at large scales.
Tested on simulated data.
Authors
Junjie Xiao, Huiwen Jia
Abstract
Modern training on a new task often starts from a previously trained model rather than from scratch, raising the question of how this initialization affects the subsequent training trajectory. Classical implicit-bias results characterize the direction selected by prolonged training, but this direction alone does not provide information regarding the intermediate behavior. We address this question through a geometric approximation of full-batch logistic gradient descent (GD) trajectories on strictly linearly separable data, with large initialization of scale $R$ motivated by prior training. From any limiting normalized initial position, we use minimum-norm projection rules to construct a unique continuous ideal path consisting of finitely many linear segments. The path has two stages: negative-margin correction followed by minimum-margin growth. We prove that, after an explicit two-stage time reparameterization, the fixed-step GD trajectory divided by $R$ converges uniformly to this path on every fixed parameter interval as $R\to\infty$. Further, our quantitative error bounds account for initialization perturbations and the transition between stages. This approximation provides asymptotic formulas for peak evaluation loss and cumulative training loss. In particular, peak evaluation loss can grow linearly in $R$ even when both endpoint losses tend to zero. The cumulative losses in the correction and margin-growth stages, normalized by $R^2$ and $R$, respectively, converge to explicit limits. Experiments on controlled geometries and fixed image features complement our theoretical results.