Selective memory in neural networks explained by reinforcement and forgetting
Learning Under Forgetting: Statistical Support-Selective Retention in Stochastic Training Dynamics
Machine LearningArtificial Intelligence
Summary
Neural networks remember some patterns in data while forgetting others, but how they choose what to keep wasn't fully understood. The authors propose a model where repeated exposure strengthens memory of recurring patterns, while forgetting acts as a filter that removes less important information. They show mathematically and with experiments how these two forces work together to keep the most meaningful data and compress information efficiently. This model provides a new way to understand how neural networks naturally focus on the most useful parts of their training data.
What this means in practice
- •For machine learning engineers: Design training schemes that strategically balance reinforcement and forgetting to improve which data patterns a model retains during learning.
- •For data compression specialists: Use insights about selective retention and forgetting to develop algorithms that compress information by preserving highly supported patterns.
Authors
Fujie Gao, Zuyue Zhang, Gang Sun
Abstract
Prior work has shown that neural networks exhibit implicit biases toward low-complexity structure (e.g., spectral bias), memorization dynamics, and compression-like effects during training, but a unified dynamical account of selective retention remains incomplete. We propose Repeated Reinforcement with Persistent Forgetting (RPF) dynamics, a minimal framework in which repeated exposure reinforces patterns and structures that recur in the data, while persistent forgetting attenuates learned information. This view treats forgetting not merely as a failure mode, but as a selection mechanism. We build the theory in three successive layers. First, in an independent-feature model, we derive an exposure-selective survival law and a support-dependent retention boundary characterizing which patterns persist under forgetting. Second, in a shared-parameter model, we show that forgetting induces spectral filtering over covariance modes, preserving strongly supported shared components while suppressing weak ones. Third, under small-step and norm/coding approximations, we show how RPF dynamics induce an implicit trade-off between data fitting and the cost of stored information, yielding Minimum Description Length (MDL)-like compression. Controlled experiments provide evidence for this reinforcement--forgetting selection mechanism in scalar memories and a nonlinear shared network. Joint reinforcement and attenuation interventions shift conditional retention, while matched exposure counts reveal forgetting-dependent effects of reinforcement timing and changes in the composition of the retained set. Together, these results show that repeated reinforcement and persistent forgetting jointly provide a controllable source of inductive bias beyond neural architecture and scale.