Optimizing covariance matrices with flexible tuning improves convergence
Optimization over covariance matrices with a parameterized metric
Machine Learning
Summary
Some math problems involve special matrices called covariance matrices, and how you measure distances between them affects how quickly you find solutions. The authors introduce a flexible way to choose this measurement by adjusting two parameters, which unifies several common methods and allows for better tuning. Their approach helps optimize these matrices more efficiently, especially when dealing with specific data. Experiments confirm that picking the right parameters can speed up solving these problems.
What this means in practice
- •For machine learning engineers: Improve training speed of models that optimize covariance matrices by tuning parameters in the optimization metric.
- •For signal processing teams: Enhance covariance matrix estimation tasks by selecting metric parameters that better condition the optimization problem.
Authors
Yibang Li, Bamdev Mishra, Pratik Jawanpuria, Cyrus Mostajeran
Abstract
The choice of Riemannian metric can strongly influence the convergence of gradient-based optimization over covariance matrices. Euclidean, Bures-Wasserstein and affine-invariant metrics are common choices, but their relative effectiveness depends on the objective. We introduce a two-parameter family defined by $X^{p}LX^{q}+X^{q}LX^{p}=U$, solved for $L$ at each tangent vector $U$, that contains all three as exact members, at $(0,0)$, $(1,0)$ and $(1,1)$, and extends past them. We treat the choice of member as a particular way of preconditioning for a given problem. To this end, we analyze the conditioning of the Riemannian Hessian at the solution. We show that it obeys a lower bound that depends on $(p,q)$ only through the exponent $r=p+q$. When the Euclidean Hessian is a pure power that mixes no eigendirections, the member $p=q=r/2$ attains that bound, and a closed-form criterion identifies the other members that do. We discuss ways to tune $r$ for a given problem. Experiments on real covariance data confirm the predicted conditioning and the benefit of tuning $r$. A task covariance example shows a further gain from tuning the shape.