SchemaMem improves long delayed memory retrieval in recurrent models
SchemaMem: Schema-Indexed Recurrent Memory for Delayed State Retrieval
Machine Learning
Summary
Remembering important information over a long time can be hard for computers that process data one step at a time. The authors created SchemaMem, a new memory system that helps computer models keep and find specific information even after many other steps happen. It uses special schemas—shared ideas—that guide how information is stored and accessed, unlike older methods. Tests showed SchemaMem remembers updated details better over long delays but needs more training. This gives insight into balancing memory retention and training effort in smart systems.
What this means in practice
- •For natural language processing engineers: Improve neural networks to better remember context across long texts for more coherent language understanding.
- •For robotics system developers: Enhance robots’ memory to recall critical past information despite long interactions or delays in sensing and acting.
Authors
Sungwoo Goo, Hwi-yeol Yun, Sangkeun Jung
Abstract
Attention provides direct access to past representations, but retaining an ever-growing history is costly. Recurrent models bound persistent state, yet must preserve selected information while processing subsequent inputs. We introduce SchemaMem, an attention-based recurrent memory architecture combining chunk-local attention with a persistent, schema-indexed phase state. Learned schema embeddings provide a shared representational reference for reading and writing. Reads use the current state, whereas writes use the layer input and static schema embeddings, excluding direct feedback from that layer's own state. Chunk-boundary commits aggregate bounded phase increments through forward computation. The same parameters also support full-history attention training before and during recurrent training. We studied selective updates, preservation, and delayed retrieval in a controlled address--value task, comparing three-layer models with approximately matched parameter counts and persistent-state dimensions. Across nine address/value settings and three training seeds, SchemaMem has higher mean written-value retention at four times the maximum training delay than both baselines, which are trained toward a higher in-range accuracy target. Updated-value recovery favors SchemaMem in all nine settings against Mamba-3 and seven against Gated DeltaNet. Defaults consistently favor Gated DeltaNet over SchemaMem at that delay, and SchemaMem requires substantially more optimization steps. These results identify a promising retention--optimization trade-off in schema-indexed recurrence.