Kalman delta networks improve memory updates in language models
Kalman Delta Networks: Uncertainty-aware Associative Memory
Machine LearningArtificial Intelligence
Summary
Language models need to remember information efficiently while reading long texts. Current methods update their memory with fixed rules that don't consider how confident they are about what they've remembered. The authors introduce Kalman Delta Networks, which keep track of both the memory and its uncertainty, allowing smarter updates that weigh new information based on reliability. This approach improves performance on language tasks compared to previous similar methods.
What this means in practice
- •For language model engineers: Build more efficient and accurate long-context language models by integrating uncertainty-aware memory updates with constant-memory linear attention.
- •For machine learning infrastructure teams: Accelerate large-scale model inference with new memory update methods offering better parallelism and uncertainty tracking within linear-attention frameworks.
Authors
Ngoc Bui, Tinglin Huang, Rex Ying
Abstract
Linear attention is increasingly used in frontier language models for efficient long-context inference and constant-memory decoding. Its fixed-size recurrent memory, however, requires an online decision at each token: what to write and how strongly to overwrite existing associations before knowing which information future queries will require. Delta-rule models learn this strength from the current token embedding but do not track confidence in the memory estimate, preventing each write from adapting to accumulated evidence. To represent this uncertainty explicitly, we reformulate recurrent associative memory as a linear--Gaussian state-space model, for which the Kalman filter is the optimal recursive estimator, and introduce a new family of models, Kalman Delta Networks (KDNs). Within KDNs, the transition propagates both the memory state and its uncertainty, allowing the Kalman gain to weight each residual write by accumulated evidence and observation reliability. Under this formulation, Delta-style updates emerge as a special case that substitutes a token-wise isotropic surrogate for predictive covariance and omits covariance tracking. Exact tracking, however, entails a dense, state-dependent Riccati recursion that is poorly suited to GPU-parallel linear-attention scans. To address this issue, we introduce two scan-compatible KDN approximations. Diagonal KDN projects each one-step posterior onto the diagonal Gaussian family through online mean-field variational inference, whereas Isotropic KDN uses an isotropic approximation with a single uncertainty scalar per head. Their uncertainty recurrences are Mobius maps, enabling associative scans with logarithmic parallel depth. Across controlled pretraining at 750M and 1.3B parameters, KDN variants consistently improve perplexity and mean downstream accuracy over state-of-the-art linear-attention models.