Summary
Many AI systems need to learn new things continuously without forgetting old ones, but doing this well is tricky. The researchers looked at how certain math tricks used for updating learning models don’t work well with a popular geometry technique called Equiangular Tight Frames (ETFs). They found that using one part of this technique, called equinormality (making classifier weights have equal length), helps models learn better when new classes are very unbalanced in number. However, using the equiangular part can actually harm learning and cause forgetting. So, making classifier weight lengths equal is the useful improvement for continual learning, especially in challenging cases with very imbalanced data.
What this means in practice
- •For machine learning engineers: Improve continual learning models on tasks with highly imbalanced class distributions by applying equinorm post-processing to classifier weights for better accuracy retention.
- •For computer vision developers: Enhance vision models that incrementally learn new object categories by balancing classifier weight norms to reduce forgetting on rare classes.
Abstract
Analytic continual learning (ACL) methods such as ACIL and GACL fit a ridge-regression classifier head on frozen features via the Sherman-Morrison-Woodbury identity, achieving exemplar-free, exact updates. A separate line of work, Progressive Neural Collapse (ProNC), shows standard continual classifiers benefit from a progressively expanding Simplex Equiangular Tight Frame (ETF) target. It is natural to ask whether fusing the two ideas, exact analytic updates and ETF targets, would yield a better closed-form continual learner. We show this is not the case as the fusion turns out to be identical to plain one-hot ACIL. Further, Simplex ETFs are not nested across class counts, so the literal "progressively expand the ETF" construction stops yielding an ETF as new tasks arrive, and it introduces catastrophic forgetting ($4\times$ that of a classic ACIL head). We therefore separate an ETF's two defining properties, equiangularity and equinormality, and apply each on its own to the classifier weights rather than the targets, where the invariance above does not hold. Equinorm post-processing of the fitted weights recovers both tail-class and overall average accuracy across the long-tailed regime from 5:1 to 100:1 imbalance (e.g. +32.0 tail-accuracy points and +16.0 average-accuracy points at 100:1 on CIFAR-100/ViT), transferring in sign across two backbones and two datasets, at a mild cost of 5.4 average-accuracy points on CIFAR-100/ViT on near-balanced streams (below $\approx 3{:}1$). Equiangular geometry, by contrast, is exactly inert on the target side but actively harmful on the weight side, usually losing to plain one-hot. Hence, the generally useful part of Neural Collapse geometry in Analytic Continual Learning is equinorm post-processing of the weights, becoming more useful as class imbalance grows; and equiangularity is at best a no-op and induces catastrophic forgetting at worst.