Large language models learn new tasks by updating specialized subnetworks
Functional Subnetwork Localization for Efficient Continual Learning in Large Language Models
Artificial Intelligence
Summary
Large language models typically forget old skills when learning new ones, which is a big challenge. The authors found special parts inside these models that handle different types of thinking without much overlap. They made a method called CiMEO that only updates these specific parts for each new task, preventing forgetting. This approach helped the model keep old knowledge and perform well on various tasks over time.
What this means in practice
- •For ai engineering teams: Enable continual training of large language models on new tasks without losing performance on prior tasks by updating only task-specific subnetworks.
- •For enterprise software developers: Maintain and adapt generative AI models for diverse business tasks by isolating updates to functional subnetworks, improving resource efficiency and stability.
Authors
Shuoran Jiang, Wentao Zhu, Yang Xiang, Dongfeng Li, Yunan Zhang, Mengchen Zhao, Yuhui Guo, Kai Chen, Youcheng Pan, Xiangping Wu, Qingcai Chen
Abstract
Biological intelligence continually adapts to new information while retaining prior skills through localized plasticity. In contrast, continual learning in large language models (LLMs) remains vulnerable to catastrophic forgetting. A promising direction is to identify functionally specialized subnetworks within LLMs that enable targeted and resource-efficient adaptation. Using gradient-attribution profiling, we localize cognition-associated functional subnetworks exhibiting extremely low mutual overlap. Building on this finding, we propose Function-isolated Memory-Efficient Optimization (CiMEO), a continual learning framework that routes each training example to its corresponding functional subnetwork, restricting parameter updates and optimizer states to selected regions. Across 1.4 million sequential instructions, CiMEO achieves the highest average performance among the compared methods on five generative benchmarks spanning programming, examination-style tasks, and mathematical reasoning, while mitigating catastrophic forgetting. Subspace-overlap and principal-angle analyses further indicate that localized updates help preserve pre-trained representations, highlighting functional subnetwork localization as a practical approach to efficient and stable continual learning in LLMs.