Proximity regularized merging improves continual adaptation of low rank adapters
When the Merge Coefficient Stops Mattering: Proximity Regularized Merging for Continual LoRA Adaptation
Machine Learning
Summary
Continually adapting AI models to new tasks without forgetting old ones is hard. The authors study a way to combine small adjustments (task vectors) into a running model more effectively. They add a gentle penalty during training that encourages these adjustments to be easier to merge, improving performance and stability. Their method helps balance learning new tasks while keeping what was learned before, making the merging process itself better.
What this means in practice
- •For machine learning engineers: Improve continual learning systems by training task vectors that merge more robustly, enhancing adaptation to new tasks without forgetting.
- •For natural language processing developers: Deploy more stable low-rank adapter merges for incremental updates to large language models on evolving datasets.
Authors
Yixuan Liu, Yuhao Sun, Sen Song, Jin Li
Abstract
Rehearsal-free continual learning with parameter-efficient adapters can be cast as a sequence of task-vector write-in operations: for each new task, a low-rank adapter is learned and merged into a running model. We propose Proximity Regularized Merging (PRM), a minimal modification to sequential LoRA merging that adds a proximal penalty during task-vector training without changing the subsequent write-in rule. PRM acts as a robust task-vector regularizer: in the reported Base->+Prox diagnostics, it improves AAA across multiple write-in rules, backbones, and class-incremental settings, while its fixed-coefficient variant remains competitive with strong coefficient-based baselines. Mechanistically, matched-prefix norm controls and proximal-strength sweeps show that proximal training shrinks the task-vector radius, lowers Fisher-weighted interference, broadens the coefficient plateau, and exposes a stability-plasticity trade-off. Together, these results suggest that the effectiveness of sequential LoRA merging depends not only on how much of a task vector is written in, but also on whether the task vector itself has been trained to be mergeable.