Bilevel spectral method improves continual learning for vision transformers
BISECT: Bilevel Spectral Decomposition for Low-Rank Continual Learning
Machine Learning
Summary
Continual learning allows AI models to learn new tasks without forgetting old ones, but it is challenging for large vision models. The authors show that existing methods that isolate task updates can limit learning new tasks by ignoring shared information between tasks. They propose a new method called BISECT that divides learning into two parts: one that retains shared knowledge and one that captures new information. This approach improves performance on several image benchmarks without increasing model size or speed requirements.
What this means in practice
- •For computer vision engineers: Improve continual image recognition models on diverse datasets without increasing inference cost by using spectral decomposition to balance shared and novel features.
- •For ai system architects: Design efficient continual learning modules for large pre-trained vision transformers that avoid interference and allow plasticity by splitting low-rank updates.
Authors
Tom Saliencro, Wen-Hsuan Liu, Rohan Desai, Amara Okonkwo, Maya Lindqvist
Abstract
Exemplar-free continual learning on pre-trained Vision Transformers increasingly relies on Low-Rank Adaptation (LoRA), yet prevailing methods govern the low-rank update space through strict subspace isolation. By projecting each incoming task into the approximate null space of historical token activations, isolation-based learners equate any cross-task feature overlap with catastrophic interference. We show that this orthogonality premise induces a structural bottleneck we term \emph{spectral starvation}: when down-projection bases are locked before task optimization, the gradient norm reaching the trainable up-projection is upper-bounded by the energy of current-task features captured by the chosen basis. Because natural visual tasks share low- and mid-level covariance structure, historical null tails contain barely $5.5\%$ of incoming task variance, starving new-class plasticity while blocking positive consolidation along shared directions. We propose BISECT (Bilevel Spectral Decomposition), a continual LoRA framework that partitions a fixed rank-$r$ budget into two orthogonal functional blocks: an invariant carrier spanned by the dominant modes of the joint historical-and-current Gram matrix, and a novelty complement extracted via generalized Rayleigh whitening followed by orthogonal complement projection. After single-task training, BISECT folds both blocks back into the frozen backbone using a closed-form precision gate that scales each rank-1 invariant atom by its current-to-joint Rayleigh ratio while preserving novelty atoms at unit gain. Across eight incremental settings on ImageNet-R, ImageNet-A, CIFAR-100, CUB-200, and DomainNet, BISECT outperforms prompt, adapter, and LoRA baselines with zero extra inference parameters or FLOPs, and remains complementary to sparse analytic expansion.