Trust region method improves constrained optimization under heavy noise
TR-SSQP: A Trust-Region Method for Constrained Stochastic Optimization under Heavy-Tailed Noise
Machine Learning
Summary
Optimizing problems with constraints can be tricky, especially when the data involved is noisy and unpredictable. The authors present a new method called TR-SSQP that helps find good solutions even when the noise is heavy-tailed, meaning unusually large fluctuations happen often. Their approach balances finding the best answer while still satisfying constraints, using special mathematical steps and momentum to keep updates stable. They prove their method reliably converges and show through tests that it works well compared to existing techniques.
What this means in practice
- •For machine learning engineers: Improve training of machine learning models with constraints under noisy data conditions prone to heavy-tailed errors.
- •For operations research analysts: Solve constrained optimization problems in uncertain environments with unstable or irregular noise distributions more reliably.
Authors
Haoxuan Wang, Yuchen Fang, Sen Na
Abstract
We consider stochastic nonlinear optimization problems with deterministic equality constraints. While unconstrained stochastic optimization is well understood, the interplay between optimality and feasibility in the constrained setting poses significant challenges. Moreover, existing theoretical guarantees for constrained stochastic methods predominantly rely on bounded-variance assumptions, leaving the heavy-tailed noise regime largely unexplored. To address this gap, we propose a novel trust-region method within the stochastic sequential quadratic programming framework, termed TR-SSQP. Our method employs a normal-tangential decomposition in the step computation to balance optimality and feasibility. In addition, we incorporate a normalization mechanism in the design of the trust-region radius, together with Polyak momentum for gradient estimation, ensuring stable updates without gradient clipping. When the trust-region radius and the momentum parameter decay at appropriate rates, we establish global almost-sure convergence of the method. To the best of our knowledge, this is the first asymptotic convergence result for constrained stochastic optimization under heavy-tailed noise. We demonstrate the promising performance of the proposed method through extensive numerical experiments, including comparisons among its variants and with existing constrained stochastic optimization methods.