New method protects language models from forgetting their knowledge when retrained
Estimating and Orthogonalizing Unknown Pre-training Gradients for Continual Fine-tuning of Large Language Models
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
When large language models learn new tasks, they often forget what they previously knew, which is a big problem. The authors propose a method called EoupCT that guesses how the original training shaped the model and then keeps new learning separate from that original knowledge. This approach helps the model remember both its earlier general understanding and new specific skills. Tests show it reduces forgetting better than other methods.
What this means in practice
- •For machine learning engineers: Maintain large language models’ general knowledge while updating them for specific tasks in real-world deployments.
- •For ai service providers: Offer more robust fine-tuned language model APIs that reduce performance loss after updates with new user data.$Commercial implications: Enables selling continually updated AI services retaining broad language understanding for diverse clients.
Authors
Bing Wang, Changchun Li, Xin-Qiang Cai, Lin Yuanbo Wu, Ximing Li, Gang Niu, Masashi Sugiyama
Abstract
Continual fine-tuning is essential for large language models (LLMs) to dynamically adapt to real-world environments, yet it inevitably suffers from catastrophic forgetting, particularly the performance degradation of previous tasks and LLMs' general-purpose knowledge. Although existing methods, such as orthogonal gradient projection, mitigate the forgetting across various fine-tuning tasks, they fundamentally fail to preserve pre-training LLMs' inherent general-purpose knowledge because the original data and gradients of off-the-shelf pre-training LLMs required by these methods are strictly unknown and highly diverse. To bridge this critical gap, we propose EoupCT, a novel framework designed to Estimate and Orthogonalize Unknown Pre-training gradients for Continual LLM fine-Tuning. Specifically, EoupCT estimates pre-training gradients by dynamically generating pseudo data that is most susceptible to forgetting for new tasks through a learnable soft prompt equipped with Gumbel-Softmax relaxation. Furthermore, we formulate a multi-objective optimization problem and introduce a first-order efficient Pareto optimizer that jointly optimizes LLM parameters and the soft prompt, rigorously enforcing orthogonality between new task updates and the estimated pre-training gradients. Extensive experiments across multiple LLMs demonstrate that EoupCT effectively preserves both task-specific proficiency and inherent general-purpose knowledge, successfully mitigating the catastrophic forgetting.