AI summaryⓘ
The authors present Neural Subspace Reallocation (NSR), a method for continual learning that saves and reuses small, compressed pieces of knowledge called LoRA modules instead of discarding them after each task. NSR stores these compressed units in a memory bank, retrieves similar past knowledge to help with new or recurring tasks, and manages which parts of the model to activate accordingly. Their theory shows that using this memory bank avoids repeated performance loss over time, and experiments confirm faster task recovery and better accuracy compared to approaches without such memory. They also find that the memory system itself, rather than complicated decision rules, is key to improved continual learning, all while keeping memory use low.
Continual LearningNeural Subspace Reallocation (NSR)Low-Rank Adaptation (LoRA)Singular Value Decomposition (SVD)Memory BankParameter SubspacesTask ReallocationBackward TransferCyclic EnvironmentsDistillation
Abstract
We introduce Neural Subspace Reallocation (NSR), which reframes continual learning as memory management over parameter subspaces. Instead of treating Low-Rank Adaptation (LoRA) modules as disposable per-task adapters, NSR manages them as compressible, retrievable memory units on a frozen backbone through a recurring cycle: (1) compress learned LoRAs via SVD, (2) reserve them in a TaskKnowledgeBank, (3) recall related past LoRAs by embedding similarity to warm-start new or returning tasks, and (4) reallocate the active subspace accordingly, with distillation protecting prior tasks. We prove that in cyclic environments any memoryless allocation policy incurs cumulative regret Omega(T(M-1)Delta_switch) relative to a history-aware policy backed by the Bank (Theorem 1). Empirically, on Split-CIFAR-100 the Bank reduces cyclic recovery time by 10x, exactly as predicted, and on the heterogeneous 5-Datasets benchmark NSR achieves the highest accuracy and the least forgetting, about 9x closer to zero backward transfer than the memoryless heuristics. Crucially, we run a controlled study that isolates which component matters: holding the Bank fixed and varying only the allocation rule, we find that a simple similarity-based retrieval rule matches or beats a learned reinforcement-learning controller (recovering recurring tasks in 0 vs 1.8 steps and reaching equal accuracy). Our central, honest finding is therefore that the memory mechanism -- compression and similarity retrieval -- rather than a learned allocation policy, drives continual-learning performance under fixed capacity. A memory-budget analysis confirms the compressed Bank stays small -- 0.29 MB of parameter memory per task -- so a top-K retention cap bounds the total footprint while preserving fast recovery for retained tasks.