LoRA initialization improves stable training with uneven learning rates
Rethinking LoRA Initialization for Robust Asymmetric Learning Rates
Machine LearningArtificial Intelligence
Summary
Fine-tuning large language models can use a method called LoRA, which relies on two connected matrix parts, each updated with different learning rates. Choosing these learning rates well is tricky because using a bigger rate on one part can cause training problems. The authors show that the usual starting setup introduces noise that makes training unstable when learning rates differ much. They suggest a new way to initialize the matrices that reduces this noise, making training more stable and easier to tune across different model sizes and settings.
What this means in practice
- •For machine learning engineers: Improve stability and reduce tuning effort when fine-tuning large language models using LoRA with asymmetric learning rates.
- •For ai model deployment teams: Enable wider safe learning rate ranges for LoRA in practical model adaptation workflows, reducing failed training runs.
Authors
Disen Liao, Yaoliang Yu
Abstract
Low-Rank Adaptation (LoRA) is often highly sensitive to learning-rate (LR) choices. Recent empirical studies suggest that, once LRs are carefully tuned, vanilla LoRA can remain competitive with many proposed LoRA variants. This shifts the central practical question from designing ever more variants to reducing the cost and brittleness of LR selection. LoRA+ addresses this issue with asymmetric LRs, using a larger LR for the up-projection matrix $B$ than for the down-projection matrix $A$. However, the ratio $η_B / η_A$ itself becomes an additional hyperparameter: large ratios can improve adaptation, but under the standard LoRA initialization they can also cause unstable training and ratio-dependent performance collapse. We identify the source of this brittleness through a signal--noise decomposition of LoRA dynamics. Under standard \initA, the update contains an initialization-induced noise term whose magnitude can be affected by the LR for $B$, $η_B$, the initialization scale $β$, and the adapter rank $r$. In practical LLM fine-tuning, this non-vanishing noise can limit the performance gains expected from LoRA+ style asymmetric LRs. To suppress this effect, we propose \initAA, a simple initialization that scales the variance of $A$ as $σ_A^2 \propto n^{-2}$ while keeping $B_0=0$. Under matched rank and training time for both initializations, this reduces the initialization-noise energy by an additional factor of $1/n$, stabilizing asymmetric LoRA training while preserving zero initial adapter output. Empirically, \initAA substantially widens the stable LR region across initialization scales, adapter ranks, and LoRA update multipliers. Across tasks and models, \initAA reduces best-ratio drift, prevents the high-ratio collapse observed with standard \initA, and makes the width-based guideline $η_B \approx n η_A$ a useful starting point for finite pretrained models.