How Much Rank Does LoRA Need? Rank-Error Bounds for Transformer Attention
2026-08-26 • Machine Learning
Machine LearningArtificial IntelligenceComputation and Language
AI summaryⓘ
The authors study how well low-rank adaptations (LoRA) can approximate changes in Transformer attention, depending on the rank chosen. They provide mathematical bounds on the smallest error possible when updating attention queries with a given rank, considering different conditions on the attention scores and input distributions. They also show how the complexity of the approximation relates to the softmax function's behavior and extend their analysis to multi-head attention and combined query/key updates. This work gives a clearer theoretical understanding of choosing the LoRA rank for adapting pretrained transformers.
LoRATransformerattentionlow-rank approximationKullback–Leibler divergencesoftmaxmulti-head attentionquery/key updatesspectral boundsapproximation error
Authors
Gerard Conangla Planes
Abstract
Choosing the rank of a low-rank adaptation (LoRA) update is usually an empirical task. In this paper, we provide a task-dependent theory of the approximation error achievable at each LoRA rank for Transformer attention. We fix a pretrained attention head, a target attention function, and a distribution over inputs from the downstream task, and bound the smallest expected Kullback--Leibler (KL) error achievable by a rank-$r$ query LoRA update. When target attention probabilities are bounded away from zero, we prove a lower bound of the error proportional to $ψ(\|d\|_2)$, where $d$ is the difference between candidate and target attention scores and $ψ(t)=\min\{t^2,t\}$. We also prove an unconditional upper bound $\min\{\|d\|_2^2/4,\sqrt2\|d\|_2\}$. Under explicit realizability, geometry, and moment conditions, we then bound the best rank-$r$ error between an explicit multiple of $ψ(\sqrt{T_r})$ and $\min\{T_r/4,\sqrt{2T_r}\}$, where $T_r$ is the downstream-weighted tail energy of the target update. We also provide target-Fisher bounds when candidate scores remain within a fixed range of the target scores, and an unrestricted lower bound when a subset of tokens carries most of the probability mass. These spectral bounds describe finite-score approximation. We then construct explicit families in which softmax saturation makes the rank required to match the attention function strictly smaller than the rank required to match the finite logits. Finally, we extend the analysis to fused multi-head LoRA and joint query/key updates, exposing the effects of rank sharing and query/key factorization constraints.