Language models keep or reshape stored information during retraining
Erased, Rerouted, or Rescaled? Post-Training and the Causal Quotient of a Language Model's Belief State
Machine Learning
Summary
When a language model is updated after being trained, some information it learned might seem less important or useful. The authors studied what happens to that old information: does the model erase it, hide it so it’s not used, or just make it smaller but still present? They found that often, this information stays inside the model but is rerouted or scaled differently depending on how training rewards certain outputs. Only when training includes weight decay or ignores that information for a long time does the model truly erase it.
What this means in practice
- •For machine learning engineers: Design post-training procedures that preserve or remove certain learned information by controlling reward functions and anchors.
- •For ai system operators: Diagnose and interpret changes in model behavior after tuning by measuring if original knowledge is erased, rerouted, or just rescaled.
Authors
Weihan Li, Tianshi Zheng, Junhao Wu, Xinlei Chen
Abstract
What happens to information a pretrained model already encodes when post-training no longer rewards using it? The common language of representation compression conflates three fates: information may be erased, rerouted away from the decision while still represented, or rescaled to occupy less variance while still represented and used. We make these fates identifiable in models whose pretraining recovers Bayesian belief states. A reward that reads only a coarse function of the hidden state defines an exact reward-null kernel. The kernel lets us separately measure whether the information remains recoverable, whether decisions causally depend on it, and how much activation variance it occupies. Theory says what is protected: KL-anchored reinforcement learning preserves the reference policy's log-odds among equally rewarded outputs, supervised and unanchored objectives carry no such constraint, and spectral compression implies neither erasure nor loss of use. In controlled worlds, post-training mostly reroutes or rescales reward-null information and leaves it decodable. Without an anchor decisions can stop using it although the representation survives, and with one they keep using it. Erasure appears only under prolonged weight decay, for distinctions that neither reward nor next-token prediction can see. Open language models show the same dissociation: in-context belief geometry stays decodable under late-layer spectral compression, and within-class behavior depends on the anchor. Post-training thus selects a causal quotient of the pretrained belief state: the reward defines decision-equivalence, the anchor and the state update protect part of what it ignores, and optimization decides whether the rest is erased, rerouted, or rescaled.