Optimizing low-parametric orthogonal matrices using Riemannian geometry tools
Riemannian Structure and Optimization for a Class of Low-Parametric Orthogonal Matrices
Machine LearningArtificial Intelligence
Summary
Some special matrices can be built by mixing blocks of simpler matrix pieces and fixed shuffles. These matrices are useful in AI because they balance being powerful and efficient. The authors study when these matrices form smooth geometric shapes called manifolds and develop math tools and algorithms to work with them efficiently without building large dense matrices. They show how their methods help optimize these matrices and fine-tune large language models more efficiently. They also look at how adding more pieces to these matrices changes their properties.
What this means in practice
- •For machine learning engineers: Implement parameter-efficient fine-tuning strategies for large language models using structured orthogonal matrices to save computation and memory.
- •For numerical linear algebra developers: Design efficient algorithms for optimizing special structured orthogonal matrices in matrix approximation problems without explicit dense matrix operations.
Authors
Ali Aliev, Maxim Rakhuba
Abstract
In this paper, we are concerned with matrices formed by block-diagonal factors interleaved with fixed permutations -- a flexible family of structured matrices. This class has recently drawn interest in deep learning architectures for its balanced expressivity-efficiency trade-off, yet efficient computational strategies for working with it remain to be found. We approach this problem through Riemannian geometry and examine under what conditions this class admits a smooth manifold structure. For the practically important case of orthogonal two-factor matrices, we derive the essential Riemannian tools and propose efficient algorithms for their implementation. The algorithms leverage automatic differentiation, support parameter sharing within each factor, and avoid explicit dense matrix construction. We test them within the Riemannian optimization framework on the best matrix approximation problem and for parameter-efficient fine-tuning of large language models. Beyond the two-factor setting, we study the geometric and matrix-theoretic properties of factorizations with a larger number of block-diagonal factors.