Ori improves language model training with smarter low-rank updates
Ori: Drift-Aware Streaming $k$-PCA for Very Low-Rank LLM Training
Machine Learning
Summary
Training large language models requires updating millions or billions of parameters efficiently. Previous approaches used a trick to simplify updates by focusing on important directions, but these directions slowly change over time. The authors developed Ori, a method that continuously tracks these shifting directions more accurately and cheaply. This leads to better training performance while using fewer resources, closing much of the gap with traditional full updates.
What this means in practice
- •For machine learning engineers: Use Ori to train large language models with reduced computation while maintaining model accuracy.
- •For natural language processing teams: Improve fine-tuning of transformer-based language models on diverse tasks using Ori's adaptive low-rank optimization.
Authors
Xinyu Luo, Jincheng Zhou, Mengbo Wang, Zilin Shen, Nadia Atallah Lanman, Petros Drineas, Bruno Ribeiro, Brian Bullins
Abstract
A prominent family of low-rank LLM optimizers, pioneered by GaLore, performs gradient-subspace projection: at each step, they project the gradient into a low-dimensional subspace and map the update back to the full model. Most existing work either recomputes the gradient subspace at certain intervals, or performs post-hoc adaptation to catch up to the moving subspace. In this work, we explicitly identify in practice the slow, stable manner in which LLM gradient subspaces drift across time. To this end, we propose Ori, a novel low-rank optimizer for training LLMs that captures the shifting principal gradient directions over time to achieve state-of-the-art accuracy under stringent low-rank constraints. Ori employs a streaming $k$-PCA algorithm inspired by Oja++, whose low per-step cost enables high-frequency, on-the-fly subspace tracking. In practice, Ori reduces the performance gap of the state-of-the-art low-rank optimizers to full-rank optimizers by up to 69.3\% on a 1B LLaMA pretraining task, and achieves the best performance on 10 out of 16 task scenarios in Roberta-Base fine-tuning. Together, these contributions allow Ori to further push the performance and efficiency frontiers of gradient-projecting optimizers, narrowing the gap to full-rank training for LLMs. Code can be found at https://anonymous.4open.science/r/Ori-Submission-2132.