Finite angular motion of learning paths for definable functions

On the gradient conjecture at infinity: Finite spherical length for gradient trajectories of definable potentials

Machine Learning

Summary

The paper shows that when using smooth and nicely describable functions, paths that follow the steepest uphill or downhill directions cannot spin around endlessly. Instead, their direction settles down over time, even if the path itself goes off to infinity. This is important because it confirms a long-standing mathematical idea about how these paths behave in complicated spaces. The authors also connect this idea to how deep learning models learn, implying that under certain conditions, the way these models update their settings will eventually point in a consistent direction.

What this means in practice

  • For deep learning engineers: Ensure that training paths of neural networks with definable smooth activations and loss functions settle into a stable direction over time.
  • For optimization algorithm designers: Use the guarantee of finite angular movement in gradient flows to improve understanding and design of optimization trajectories in definable settings.

A theory result. No direct application yet.

Authors

Thang Do, Arnulf Jentzen, Benno Kuckuck

Abstract

We prove that for every open $ U \subseteq \mathbb R^n $ the spherical projection of every gradient trajectory $θ(t)$, $t \geq 0$, of a definable $C^1$-potential $F \colon U \to \mathbb R$ has finite length and, in particular, converges (directional convergence). In other words, every gradient trajectory of a definable $C^1$-potential has finite total angular motion. Here, definable means definable in an arbitrary o-minimal structure expanding the real field. Notably, the gradient trajectory in this result is an arbitrary solution of the gradient flow differential equation (whose solutions may not be unique) with no further assumptions: The gradient trajectory may be unbounded and diverge to infinity or it may be convergent. The established theorem therefore provides affirmative answers to both the gradient conjecture at infinity for definable potentials and the definable Thom gradient conjecture. We apply our theorem to the training of deep neural networks (DNNs) to conclude that, in the case of definable and continuously differentiable activation and loss functions, the spherical projection of the DNN training flow must always converge (directional convergence).