Parameter gradient flows close only under specific model rank conditions

When Parameter Gradient Flows Close in Representation Space

Machine Learning

Summary

Gradient flows describe how machine learning models change their parameters to improve performance. This paper studies when these flows behave predictably based on the model's internal structure, focusing on matrix factorization models. The authors find that only certain rank conditions of these factors ensure stable behavior called closure, while others are excluded. They also provide insight into when specific loss functions cause or fail this closure, improving understanding of model training dynamics.

What this means in practice

A theory result. No direct application yet.

Authors

Shengye Tao

Abstract

The represented gradient flow of a parameterized model depends on a mobility that can retain hidden parameter information. We study when one entire represented mobility map works for every autonomous loss from one fixed initialization. For two-factor matrix representations wider than both output dimensions, with individually full-row-rank factors, we classify all admissible joint ranks: closure occurs precisely at effective scalar-balanced minimal joint rank or a two-value signed spectral family at full joint rank. All intermediate ranks are excluded; positive represented flows belong to known square-root families. For general representations with constant ambient representation and gradient-Lie ranks, ordinary closure is equivalent to a local invariant-level metric in the analytic case and specified smooth regimes. An endpoint section forces the initially unrestricted mobility map to admit a locally smooth representative on the represented image. We also separate testing families: linear losses locally admit closure at every submersion, whereas one fixed translated strongly convex quadratic family fails it on a delayed-bracket example. A fixed output readout gives an ordinary squared-loss witness. These supporting results do not characterize quadratic-only closure in general. We retain the unresolved smooth output-rank-two case with Lie rank at least four. A frozen deterministic experiment checks a separate engineered-loss construction; it is distinguished from the squared-loss proof, universal theorems and empirical learning performance.