The Stable Recovery Manifold: Geometric Principles Governing Recoverability in Continual Learning
2026-06-11 • Machine Learning
Machine Learning
AI summaryⓘ
The authors studied how deep learning models forget old tasks when learning new ones, focusing on the geometry of how knowledge is stored and recovered. They found that even though the model's internal representations change a lot, the core way to recover old knowledge stays stable and compact. This suggests forgetting is more about how accessible or aligned the knowledge is, rather than it being completely lost. Their work challenges the idea that forgetting destroys information, showing it can be recovered from stable subspaces.
catastrophic forgettingcontinual learningrepresentational driftrecovery subspace dimensionalitymanifold alignmentsingular value decompositionsplit CIFAR-100ResNet-18principal anglesprobe performance
Authors
Ayushman Trivedi, Bhavika Melwani
Abstract
Catastrophic forgetting is often viewed as the destruction of previously learned knowledge during sequential learning. Building on the Accessibility Collapse framework, we investigate the geometric structure of recoverability in continual learning. Using Split CIFAR-100 and a sequentially trained ResNet-18, we analyze recoverability, representational drift, and recovery complexity across ten tasks. We introduce Recovery Subspace Dimensionality (k_t), a measure of the minimum number of singular directions required to preserve 90 percent of full probe performance. Contrary to our Recoverability Diffusion hypothesis, recovery dimensionality remains stable throughout training (mean k_t = 8.0) despite substantial representational drift. Principal-angle drift strongly predicts recoverability (r = -0.862), and a simple geometric model explains 82.2 percent of recoverability variance. These findings support the Stable Recovery Manifold hypothesis, suggesting that forgotten knowledge remains compactly decodable despite representational reorganization. The results indicate that catastrophic forgetting is primarily an accessibility and manifold-alignment problem rather than information destruction.