Language model forgetting includes reversible shifts and lasting losses
When Forgetting is not Catastrophic: On the Mechanics of Spurious Forgetting
Computation and LanguageMachine Learning
Summary
When language models are fine-tuned on new information, they can seem to forget old facts. The authors studied this and found that sometimes this forgetting isn’t really permanent—it’s a hidden shift that can reverse itself before any real loss happens. They explain this by showing how the model’s memory shifts in a shared way and then slowly loses some specific facts. This means that some forgetting is temporary and others are lasting, depending on how new information interacts with old memories.
What this means in practice
- •For machine learning engineers: Improve fine-tuning techniques by distinguishing reversible forgetting from permanent loss to better preserve old knowledge.
- •For ai system maintainers: Use understanding of memory shifts to monitor and recover information in deployed language models during updates.
Authors
Vedant Palit, Florent Draye, Nicolas Zucchet, Zhijing Jin, Bernhard Schölkopf
Abstract
Knowledge that a language model appears to forget during finetuning often remains stored and can be recovered, a phenomenon called spurious forgetting. Finetuning on new facts can even produce forgetting that undoes itself: recall of the old facts collapses, recovers as training continues on new facts alone, and only then erodes for good. We seek to understand when such forgetting is not catastrophic. A minimal associative memory reproduces these dynamics with three ingredients: keys with shared structure, concentrated new values, and normalization in the network. Finetuning moves all old representations along a common direction, hiding the old facts while preserving their relative geometry; normalization withdraws this shift once the new facts are learned, whereas fact-specific changes accumulate and cause the erosion. Moreover, subtracting the common shift eliminates the collapse in a Transformer trained on synthetic data, and removing a single direction from each weight update restores old facts in a pretrained language model. Forgetting thus combines a shared, reversible loss of access with a slow erosion of individual facts, and only the second is catastrophic. Which one dominates depends on whether the new data move old memories together or apart.