Parameter-efficient fine-tuning methods show subtle adaptation and retention differences
Are Parameter-Efficient Fine-tuning Methods Really Different?
Machine Learning
Summary
Fine-tuning large AI models with fewer new parameters can be done in several ways, but it's unclear how these methods differ. The authors compare six such methods and find that many keep key properties of the original model, even if not explicitly designed to do so. Some methods trade off adapting well with remembering previous knowledge differently. They also find that changes in certain parts of the model's mathematical structure influence performance more than others. This work helps understand how different tuning approaches affect AI models' behavior.
What this means in practice
- •For machine learning engineers: Choose among parameter-efficient fine-tuning methods that balance learning new tasks and retaining existing knowledge based on their different retention and adaptation profiles.
- •For ai model deployment teams: Manage updates to large pretrained models by restoring key spectral properties to maintain task performance after fine-tuning adjustments.
Authors
Yikuan Li, Pinyan Lu, Fanghui Liu
Abstract
Parameter-efficient fine-tuning (PEFT) offers many parameterizations, yet their methodological and functional differences remain unclear. We compare six methods in language and diffusion models to examine how their parameterizations relate to task performance, forgetting, and changes in pretrained weight geometry. Motivated by the spectrum-preserving design of orthogonal fine-tuning (OFT), we first ask whether spectral preservation is itself important for adaptation and retention. We find that the selected LoRA-family methods also approximately preserve pretrained geometry, and that restoring their slightly drifted singular-value spectra largely preserves task performance, questioning the necessity of explicit geometric preservation. Beyond this, we observe that some methods exhibit distinct adaptation--retention trade-offs that vary across settings: LoRA most consistently limits forgetting at competitive performance, DoRA achieves higher mean task scores than LoRA in most comparisons, while PiSSA often incurs greater retention costs. Further intervention experiments suggest that while performance gains from different PEFT methods can be attributed to modifications in different groups of spectral components, we consistently find that restoring dominant rather than intermediate or trailing components produces the largest mean reduction in general-text NLL or base-image drift. Together, these results motivate evaluating geometric constraints through their functional consequences rather than preservation alone. Code is available at https://github.com/Kuaaannn/PEFT_methods.