Convergence of Continual Learning in Homogeneous Deep Networks
2026-06-29 • Machine Learning
Machine Learning
AI summaryⓘ
The authors study how certain types of machine learning models learn tasks one after another, focusing on models that are homogeneous (scale in a particular way). They describe this learning process as a series of steps projecting onto sets defined by each task's margin. They find that learning doesn't always perfectly converge, but under some conditions related to the model's structure and the order of tasks, it can reach good solutions reliably. The authors also expand their findings to cover models that do regression tasks, not just classification.
homogeneous modelscontinual learningclassificationregressionnonconvex projectionlocal convergencetask margindeep networkssequential projectionregularization
Authors
Matan Schliserman, Gon Buzaglo, Itay Evron, Daniel Soudry
Abstract
We characterize weakly regularized continual classification in homogeneous models as sequential projections onto task margin sets. This result generalizes prior analyses restricted to either stationary (single-task) deep models or continual linear models. We show that global convergence generally fails, even for simple models linear in data but nonlinear in parameters. Nevertheless, by leveraging results from nonconvex projection theory, we identify regularity properties of homogeneous deep networks that guarantee local linear convergence under random and cyclic task sequences. Finally, we extend our analysis to continual regression, unifying the framework for homogeneous models.