Visual continual learning improves with efficient adapter updates
RASP: Read-All, Sparse-Plasticity Continual Learning in a Stationary-Size LoRA
Machine Learning
Summary
When teaching computers to recognize images over time, they often forget old lessons when learning new ones. The authors found a way to keep a small, fixed system of parameters that can learn multiple image tasks without forgetting past knowledge. Their method reads all features fully when recognizing images but only updates a small, important part when learning new information. This approach keeps the system size steady while matching or beating other methods that grow larger with each task.
What this means in practice
- •For computer vision engineers: Build image recognition systems that learn multiple tasks over time without increasing model size or losing old knowledge.
- •For machine learning platform teams: Deploy memory-efficient continual learning adapters that maintain model performance across varied image datasets with fixed storage budgets.
Authors
Wen-Hsuan Liu, Amara Okonkwo, Tom Saliencro, Rohan Desai, Maya Lindqvist
Abstract
Parameter-efficient continual learning on pre-trained Vision Transformers largely relies on a costly premise: preventing catastrophic forgetting requires either spawning a dedicated LoRA branch per session---causing adapter parameters to scale linearly with task horizon---or applying sparse Mixture-of-Experts gating symmetrically to both forward and backward passes, which sacrifices representational rank at test time. We argue that reading consolidated knowledge and writing new task credit are fundamentally asymmetric operations. Because visual features in a shared low-rank space distribute across rank-1 atoms rather than localizing in disjoint task silos, forward inference should read all atoms at full rank capacity, whereas backward optimization should restrict plasticity to a sparse subset of input-aligned channels. We introduce RASP (Read-All, Sparse-Plasticity LoRA), which adapts a single stationary-size adapter across sequential tasks. At each step, a scene-modulated cosine router selects the top-$q$ unlocked atoms to receive gradients, blocks updates on the remaining $r-q$ channels, and penalizes Frobenius Gram coherence between active and non-updated output directions. Across sessions, the router's empirical activation trace and each atom's exact spectral norm form a zero-overhead geometric consolidation index $ω_j = \sqrt{\tilde{c}_j \tildeσ_j}$ that drives Cauchy step attenuation and boundary anchor locking. Across CIFAR-100, CUB-200, ImageNet-R, and ImageNet-A, RASP matches or surpasses branch-expanding LoRA learners under a strictly constant parameter footprint.