New method reduces forgetting in sequential learning tasks
SPACE-LoRA: Allocating Activation-Subspace Protection for Continual Learning
Machine LearningArtificial Intelligence
Summary
When computers learn new tasks one after another, they often forget what they learned before, a problem called catastrophic forgetting. The authors propose a method called SPACE-LoRA that helps protect important parts of the model related to old tasks by limiting changes along critical directions. This method decides how much protection each part needs based on how sensitive it was to earlier tasks, helping maintain past knowledge while still learning new things. They tested this approach on various tasks and showed it can reduce forgetting effectively.
What this means in practice
- •For machine learning engineers: Improve multi-task learning systems by reducing forgetting when models learn tasks sequentially.
- •For software developers integrating adaptive ai: Maintain stable AI behaviors in applications that require ongoing updates without retraining from scratch.
Authors
Seunghyun Yoo, Kiseok Kim, Hyeontae Joo, Junyeop Bang, Hwangnam Kim
Abstract
This study addresses the catastrophic forgetting problem that occurs when sequentially learning successive tasks using Low-Rank Adaptation (LoRA) from a lifelong learning perspective. While existing approaches have primarily constrained parameter updates or learning subspaces to reduce interference with past knowledge, they have not fully considered additive interference. This occurs when a newly added residual adapter on top of a fixed past model generates non-zero responses along input directions important for old tasks, thereby altering previous predictions. To this end, we propose Subspace Protection with Allocated Capacity for Efficient Continual Adaptation (SPACE-LoRA). SPACE-LoRA directly suppresses the responses of the new residual branch along input activation directions that are important for old tasks and adaptively determines the protection coverage for each module based on past-task sensitivity estimated via a common Fisher sensitivity-based coverage target. Under a fixed LoRA rank, this approach adaptively adjusts module-specific protection coverage while suppressing interference along input directions sensitive to old tasks. We assess the effectiveness of activation-subspace protection in mitigating catastrophic forgetting and examine the role of sensitivity-guided protection in continual learning across diverse tasks. Code is available at https://anonymous.4open.science/r/SPACE-LoRA-7864.