Bayesian LoRA method estimates priors for better model fine-tuning
Prior Predictive Matching for Bayesian LoRA
Machine Learning
Summary
Fine-tuning large language models often requires guessing certain settings, called priors, which affect how well the model adapts and estimates uncertainty. The authors propose a new way to estimate these priors before training, using mathematical analysis of how small updates change model outputs. Their method works across different sizes of model updates and improves performance measures like accuracy and uncertainty calibration without needing extra validation data. Experiments show their approach matches true best settings and helps with detecting unusual inputs.
What this means in practice
- •For machine learning engineers: Set hyperparameters for Bayesian LoRA fine-tuning of large language models without needing extra validation data.
- •For nlp model developers: Improve uncertainty estimates and out-of-distribution detection when adapting pretrained language models with low-rank updates.
Authors
Tomasz Kuśmierczyk, Eliezer de Souza da Silva, Marcin Sendera
Abstract
Bayesian variants of Low-Rank Adaptation (LoRA) provide principled uncertainty estimation when fine-tuning large foundation models, but require specifying the prior over the low-rank weight updates. We address this hyperparameter selection problem by introducing a closed-form prior-predictive estimator that operates before fine-tuning. We analyze a second-order expansion of the network and quantify how the low-rank prior drives function-space deviations from the pretrained outputs. We also show that these prior-predictive quantities depend on the prior only through the product of the adaptation rank and the fourth power of the prior scale. Because the training targets share a single draw of the latent weight update, we can leverage the covariance between example residuals to derive a moment-matching estimator. Furthermore, our theoretical analysis establishes that the optimal prior precision scales as $r^{-1/2}$ with the rank $r$, allowing a single estimate to generalize across all ranks. Controlled experiments across regression and classification tasks validate our approximation and show that our estimator recovers the true prior scale. On RoBERTa-large, our approach improves predictive negative log-likelihood (NLL) and calibration over both MAP and marginal-likelihood selection without requiring validation data, while achieving competitive out-of-distribution detection.