Finite-horizon memory efficiency bounds in noisy recurrent systems
Finite-Horizon Fisher Memory in Two-Sided Power-Bounded Recurrent Systems
Machine Learning
Summary
This work explores how information is stored and retained in noisy memory systems that repeat signals over time. The researchers show that while certain types of memory systems can redistribute information, they cannot increase its overall average amount. They develop mathematical bounds on how memory fades with time and demonstrate ways to preserve information even when noise is present. They also validate their findings by training systems to nearly optimally remember and recall stored information.
What this means in practice
- •For machine learning engineers: Design recurrent neural networks with provable memory retention guarantees under noise to improve time-series prediction accuracy.
- •For signal processing teams: Implement finite-horizon recurrent memory systems optimized to maintain information fidelity despite noisy signal contamination.
Authors
Jeonghoon Lee
Abstract
We analyse allocation, admission and post-write retention in finite-horizon linear-Gaussian noisy recurrent memories. At every horizon, the directional Fisher memory $M_n$ satisfies $\operatorname{tr}M_n=N$: non-normality redistributes information but cannot raise its spherical average, while normal carriers satisfy $M_n=I$. For bi-power-bounded carriers, we derive uniform $1/n$ lag bounds, identify the limit of $M_n$ with the inverse of the classical Cesàro asymptotic limit of $W^\top$, and give finite-horizon error bounds. A time-varying coupling defines an end-to-end store operator. The writer-optimal direction need not be store-optimal. After writing ends, an invertible hold preserves the full stored Fisher matrix. Additive contamination bounded by $α$ times the closure covariance retains at least $1/(1+α)$ of that matrix; a covariance-aware decoder attains the corresponding accuracy. With recurrent carriers held fixed, training input masks and linear readouts approached the task-specific optimum in 160 runs, with median normalized Rayleigh efficiency above $0.998$. Binary accuracy matched the Gaussian prediction to mean absolute error below $0.002$ over more than four orders of magnitude in $J$. In a separate pre-specified study of 320 runs, trained masks followed the designated input-time objective in both carrier types, in 16 of 16 draws. These studies used development-seen carriers and are pre-specified validations, not blind holdouts. The same fixed design reproduced the objective-specific result in 16 of 16 draws on carriers unused before run commitment. Exact isolation preserved information, while a decoder fixed at its training horizon fell to chance; inverse-adjoint transport restored its sampled decisions to numerical precision.