Optimization behavior differs despite same function representations
Optimization Geometry of Equivalent Brownian RKHS Representations
Machine Learning
Summary
Sometimes the same function can be described in different ways that look different on the surface but actually represent the same thing. The authors study these different ways, called nodal, increment, and spectral coordinates, in a specific mathematical setup involving Brownian motion and function spaces. They show how these different descriptions lead to different paths when using optimization, even though the underlying function and problem stay the same. This helps understand why certain optimization methods behave differently depending on how you set up the problem.
What this means in practice
- •For machine learning engineers: Choose coordinate parameterizations to improve optimization stability and efficiency without changing model function or regularization.
- •For numerical analysts: Understand how different finite parameterizations affect algorithmic convergence and conditioning in function approximation problems.
Authors
Mahdi Mohammadigohari, Gustau Camps-Valls
Abstract
Equivalent finite parameterizations can represent the same functions and intrinsic norm yet induce different optimization algorithms. We study this effect in a controlled finite Brownian RKHS with nodal, increment, and spectral coordinates. Classical finite-element, RKHS-interpolation, Brownian-covariance, and mixed-boundary DCT identities make the shared hypothesis class, Brownian energy, approximation operator, and coordinate maps explicit. Our main results concern the optimization geometry of this fixed model. With mapped initialization, identical scalar steps, and identical minibatches, nodal and spectral GD/SGD have exactly the same mapped trajectories. Increment GD is an explicit Euler step for the constant Brownian/Sobolev metric, with factor $1/h$. For Brownian-regularized least squares, $κ_2(\mathbf H_{\mathrm{inc}})\le1+A/ρ$, independently of grid resolution $G$ for fixed $A$, $ρ>0$, and the stated normalization. Under the stated standard-Adam convention, the universal orthogonal equivariance group is exactly the signed permutations; the block DCT-VIII transform is not one. Float64 tests over five grids numerically verify the finite identities, mapped one-layer and recursive trajectories, conditioning predictions, and theorem-matched Adam separation. Thus coordinate effects are isolated without changing the represented functions, intrinsic regularizer, or approximation space.