Fine-Tuning Regimes Define Distinct Continual Learning Problems
2026-04-23 • Machine Learning
Machine Learning
AI summaryⓘ
The authors studied how machines learn tasks one after another without forgetting old ones, focusing on how much of the model is allowed to change during learning. They found that the amount of the model being updated (called trainable depth) affects which learning method works best, meaning different methods rank differently depending on this setting. They also observed that allowing deeper updates leads to bigger changes in the model and more forgetting of previous tasks. Their work suggests it's important to consider these training settings when comparing continual learning methods.
Continual LearningFine-tuningTrainable DepthTask Incremental LearningCatastrophic ForgettingProjected OptimizationEWC (Elastic Weight Consolidation)LwF (Learning without Forgetting)SI (Synaptic Intelligence)GEM (Gradient Episodic Memory)
Authors
Paul-Tiberiu Iordache, Elena Burceanu
Abstract
Continual learning (CL) studies how models acquire tasks sequentially while retaining previously learned knowledge. Despite substantial progress in benchmarking CL methods, comparative evaluations typically keep the fine-tuning regime fixed. In this paper, we argue that the fine-tuning regime, defined by the trainable parameter subspace, is itself a key evaluation variable. We formalize adaptation regimes as projected optimization over fixed trainable subspaces, showing that changing the trainable depth alters the effective update signal through which both current task fitting and knowledge preservation operate. This analysis motivates the hypothesis that method comparisons need not be invariant across regimes. We test this hypothesis in task incremental CL, five trainable depth regimes, and four standard methods: online EWC, LwF, SI, and GEM. Across five benchmark datasets, namely MNIST, Fashion MNIST, KMNIST, QMNIST, and CIFAR-100, and across 11 task orders per dataset, we find that the relative ranking of methods is not consistently preserved across regimes. We further show that deeper adaptation regimes are associated with larger update magnitudes, higher forgetting, and a stronger relationship between the two. These results show that comparative conclusions in CL can depend strongly on the chosen fine-tuning regime, motivating regime-aware evaluation protocols that treat trainable depth as an explicit experimental factor.