Local support learning helps large models remember old skills during new training
Local Support Learning
Machine LearningArtificial Intelligence
Summary
Large AI models often forget what they learned before when trained on new tasks, a problem called catastrophic forgetting. The authors found that viewing this issue geometrically led them to create Local Support Learning (LSL), which keeps a model’s old skills by using two parts: one that learns new tasks and another that only activates this learning on familiar data. This gate is based on a model that knows what data it has seen, helping avoid disturbing previous knowledge. Their approach works on models with billions of parameters without needing to save old data and uses computing resources efficiently.
What this means in practice
- •For ai model developers: Maintain pretrained capabilities when fine-tuning large language models on new data without storing previous training data.
- •For software engineers: Improve update efficiency and robustness of AI models deployed in evolving scenarios by locally restricting new training effects.
Authors
Assaf Ben-Kish, Akarsh Kumar, James Glass, Raja Giryes
Abstract
We explore catastrophic forgetting in the context of large pre-trained models. By considering forgetting as a geometric problem in the input space of each weight matrix, we uncover a natural retention objective under which updates produced by gradient-based optimizers are suboptimal. Following this observation, we propose Local Support Learning (LSL), a general-purpose framework that augments gradient-based training for retention of prior capabilities without access to prior data. During a new learning phase, LSL pairs two components with distinct roles: a standard weight adapter, trained as usual to minimize the loss, and a gating function that enables the adapter only on input activations from its own training distribution, making the update local to that distribution. The key challenge is that this gate must route data from all learning phases while training only on data from the current one. We address this with a gate based on a Gaussian Mixture Model (GMM), whose likelihood decays rapidly away from its training data, giving it a natural tendency to stay closed on data from prior phases. We show that this post-training approach can resolve forgetting in LLMs of up to 7 billion parameters, retaining both pretrained and finetuned capabilities across multiple training phases, while being efficient in memory and compute, robust to hyperparameter choice, and showing scaling potential.