TailLoR: Protecting Principal Components in Parameter-Efficient Continual Learning
2026-06-04 • Machine Learning
Machine Learning
AI summaryⓘ
The authors introduce TailLoR, a method that updates neural network weights in a smart way by focusing on small changes in a special matrix called the singular value matrix. They use fixed components (U and V) from the original network as a stable reference to guide these updates. TailLoR discourages changes in dominant directions, which helps prevent interference with what the network already knows, while allowing detailed adjustments in less dominant directions. This approach aims to improve continual learning by efficiently adapting the model without forgetting.
Continual LearningParameter-efficient FinetuningSpectral DecompositionSingular Value Decomposition (SVD)Singular ValuesLow-rank UpdateNeural NetworksInterferenceAdaptation
Authors
Marius Dragoi, Ioana Pintilie, Alexandra Dragomir, Antonio Barbalau, Florin Brad
Abstract
Parameter-efficient finetuning methods based on spectral decomposition have enabled progress in Continual Learning. In this paper we introduce TailLoR, which utilizes the singular bases U and V of the pre-trained weights as a fixed reference frame to learn a low-rank update applied to the singular value matrix. A soft spectral penalty discourages updates aligned with dominant singular directions, reducing interference while routing fine-grained adaptation into the highly flexible, long-tail spectral coordinates.