Finite particle approximation tracks complex gradient flows over time

Finite-Sample Approximation of Hessian-Guided Perturbed Wasserstein Gradient Flows

Machine Learning

Summary

Gradient flow methods help improve solutions by gradually moving toward better answers, even when those answers are complex distributions rather than simple points. This paper studies a version of these methods that adds random Gaussian noise to help avoid getting stuck in difficult spots called saddle points. The authors analyze how well a finite number of particles can approximate this noisy gradient flow over longer periods, especially when the curvature of the problem's landscape changes over time. They show how some patterns of curvature—negative followed by positive—can let the approximation stay accurate even when it temporarily becomes unstable, and they validate their findings on specific mathematical models.

What this means in practice

  • For machine learning engineers: Predict stability and accuracy limits when simulating particle-based gradient flows with noise for training complex nonconvex models.
  • For computational mathematicians: Use curvature-informed error bounds to design longer reliable simulations of noisy gradient flows in high-dimensional probabilistic systems.

A theory result. No direct application yet.

Authors

Ryotaro Kawata, Atsushi Nitanda, Taiji Suzuki

Abstract

Wasserstein gradient flow extends gradient descent to probability measures. Its Hessian-guided perturbed variant (PWGF) adds Gaussian perturbations to escape saddle points in nonconvex problems. We investigate when its approximation by finitely many interacting particles remains accurate over growing time horizons. Our analysis retains the curvature accumulated along the population-driven reference path: negative curvature can amplify approximation errors, while subsequent positive curvature can damp their influence. This captures favorable scenarios in which temporary instability is compatible with accurate tracking over growing horizons. Under regularity assumptions and a prescribed common perturbation schedule, we prove particle and objective-value tracking bounds on a high-probability event for reference paths satisfying explicit conditions on accumulated curvature. To handle state-dependent Gaussian jumps, we construct a population-first coupling that preserves the reference particles' conditional independence and reduces jump errors to covariance comparison. We verify the conditions in a variance-plus-cosine model, where curvature recovery yields a growing-horizon tracking guarantee. We also establish local attraction, transverse descent, and positive second variation in two regions of a regularized matrix-factorization model, motivating a positive-negative-positive curvature pattern.
View PDFOpen arXiv