Optimizer memory can cause language models to forget old answers
A Step Towards Forgetting: Optimiser History and the Loss of Answer Mass
Computation and LanguageArtificial Intelligence
Summary
When a language model is fine-tuned, it sometimes becomes less likely to give correct answers it previously knew, even though the immediate training signals try to keep those answers. The researchers found that the history stored in the optimizer, which remembers past training steps, can push the model away from those old answers. They analyzed how the optimizer’s past gradients affect forgetting and showed that older updates tend to cause more forgetting while recent updates help maintain knowledge. Resetting the optimizer’s memory can reduce forgetting and help the model remember better.
What this means in practice
- •For machine learning engineers: Adjust optimizer configurations during fine-tuning to reduce loss of previously learned answers in language models.
- •For ai system operators: Use momentum reset methods to maintain knowledge retention when updating deployed language models.
Authors
Valeria Ruscio, Seth Nabarro, Keiran Thompson
Abstract
During fine-tuning, a language model can assign less probability to previously learned answers even when the current gradient acts to preserve that probability. With momentum, each update also carries gradients computed at earlier model states, and these stored contributions can push the model in the opposite direction. We investigate how this optimiser memory contributes to forgetting by separating old-task loss into confusion among its answers and leakage of probability outside the answer set. Across three language-model families, answer mass consistently declines while discrimination among old answers usually improves: the model becomes less likely to produce answers that it can still distinguish correctly. Decomposing Adam updates reveals opposing contributions to this loss of answer mass. Over training, accumulated history favours leakage, while the current gradient opposes it. Resolving history by age shows that the harmful contributions come mainly from older gradients of the new task, whereas recent gradients tend to protect the old answers. Changes in history's effect are dominated by its orientation relative to the old-task gradient. Interventions that reset momentum while matching the initial update norm establish that stored history affects retention, with state-dependent immediate effects and lower final old-task loss over longer Adam continuations, mainly through recovered answer mass. Finally, integration along finite updates shows that most sampled large loss increases are captured by local projections, while curvature along history amplifies some events. Together, these findings reveal how an optimiser's memory can erode learned behaviour even as its current gradient acts to preserve it.