PoLoRA: A Preconditioned Orthogonalized LoRA Optimizer
2026-07-20 • Machine Learning
Machine LearningComputation and Language
AI summaryⓘ
The authors study ways to improve LoRA, a method that fine-tunes large language models more cheaply by adding small, low-rank changes to the model's weights. They find that popular optimizers like Adam do not fully use the special matrix structure of LoRA, and other proposed methods don’t consistently help either. To fix this, the authors propose PoLoRA, a new optimizer that understands the product structure of LoRA and adjusts updates carefully based on the model's loss. Their tests show PoLoRA trains models faster and with less sensitivity to parameter settings than Adam, adding only a small computational cost.
Low-rank adaptation (LoRA)Adam optimizerMatrix factorizationPreconditioningSpectral updateInstruction tuningLoss functionLearning rateLarge language modelsFine-tuning
Authors
Nikhil Ghosh, Tetiana Parshakova, Robert M. Gower
Abstract
Low-rank adaptation (LoRA) makes finetuning large language models cheaper by adding to each weight matrix a trainable low-rank update parameterized as the product of two matrices. These matrices are usually trained with Adam, which treats them as a single flat vector of parameters and ignores both the matrix and product structure of LoRA. Applying a matrix-aware optimizer such as Muon to each factor does not consistently improve over Adam, and neither do the product-aware Muon variants proposed in concurrent works. To realize consistent gains, we introduce PoLoRA, a Preconditioned Orthogonalized LoRA optimizer built from three ingredients: a product-aware spectral update direction, curvature preconditioning derived from controlling the per-sample loss change, and a magnitude rule that controls the sizes of both the factor and merged updates. We evaluate PoLoRA on instruction-tuning datasets for code and math across models from 1B to 8B parameters, and find that it reaches the final held-out loss achieved by tuned Adam in 1.2-1.7 times fewer steps, while adding at most 3% per-step overhead. Compared to Adam, PoLoRA is also less sensitive to the learning rate, and its optimal learning rate is stable across ranks.