Normalized Low-Rank Adaptation
2026-08-31 • Machine Learning
Machine Learning
AI summaryⓘ
The authors study a method called LoRA that helps adapt large models efficiently but can be unstable during training. They noticed that the early steps of training depend mostly on one part called the down-projection. To fix this, they propose NoRA, which normalizes this down-projection to make training more stable and faster. They also find that just normalizing once at the start improves LoRA without extra work during training. Their method helps models learn better and forget less, and it does not add complexity or slow down the model when used.
Low-Rank Adaptation (LoRA)model adaptationoptimization dynamicsdown-projectionnormalizationtraining stabilitycatastrophic forgettingpretrainingfine-tuningreinforcement learning
Authors
Jiale Kang, Ziyin Yue, Zheng Zhan, Yangyi Huang, Weiyang Liu
Abstract
While low-rank adaptation (LoRA) is widely used for parameter-efficient model adaptation, how to regularize its training dynamics for stable and effective optimization remains underexplored. Because LoRA initializes the up-projection to zero, its early optimization dynamics are largely governed by the down-projection. Building on this observation, we introduce Normalized Low-Rank Adaptation (NoRA), a simple yet effective method that normalizes the down-projection matrices during training. We further show that the same normalization can be applied only at initialization, improving standard LoRA without requiring repeated normalization throughout training. Across pretraining, supervised finetuning, and reinforcement learning, NoRA consistently accelerates convergence, improves performance and training stability, and mitigates catastrophic forgetting. These benefits require neither additional trainable parameters nor inference-time computation, making NoRA a simple and broadly applicable enhancement to LoRA.