Kathleen Remembers: Length-Invariant One-Shot Recall Without Attention
2026-08-31 • Computation and Language
Computation and LanguageMachine Learning
AI summaryⓘ
The authors address a problem where models that remember sequences struggle to recall single past events exactly when they happened a long time ago. They add a special memory component called a "notebook" that can store and retrieve information accurately regardless of how far back it was seen. This notebook allows the model to selectively forget specific facts and identify where each piece of recalled information came from. They show their notebook helps the model remember rare repeated words in text better over long distances, even beyond the training length. Their experiments were carefully done and can be reproduced on standard hardware.
recurrent sequence modelsholographic associative memoryone-shot recallattention headswrite-triggered forgettingselective unlearningtoken attributionWikiText datasetzero-shot generalizationlogits
Authors
George Fountzoulas
Abstract
Recurrent, attention-free sequence models share a structural weakness: a fading state cannot perform exact recall of something seen once, far in the past. We add to the Kathleen trunk a second memory layer -- a "notebook": a fixed-key holographic (HRR) associative store with a learned local write gate, a self-gating raw read, and write-triggered forgetting -- 25K parameters that attach to the logits of any trunk. (1) Mechanism: on a controlled needle-in-haystack task the notebook reaches 80-82% one-shot recall at 4x the training length, where the bare trunk scores ~4% and a parameter-matched attention head scores 100% inside its training length and 0% beyond it. Addressing is length-invariant by construction; the untrained memory alone recalls at 90% accuracy identically at 512, 2048 and 4096 bytes. Because the store is a linear superposition, two capabilities follow from arithmetic alone: selective unlearning (one subtraction erases one fact to chance, retained facts unharmed) and per-token attribution (counterfactual erasure names the source fact of every correct byte, 100% provenance). (2) Real text: on WikiText-2 bytes the notebook improves prediction of repeated rare words by +0.15-0.27 bits/byte, the gain growing with the distance between mentions and holding zero-shot at 4x training length; write-triggered forgetting eliminates memory pollution at 8x length (first-mention cost +0.33 -> -0.004). (3) Scope and scale: a parameter-matched attention head does generalize on natural-text repetition, so the notebook's claim is exact recall at O(L); on a WikiText-103 ladder (8 to 512 MB) the zero-shot repeat gain rises monotonically. All experiments are pre-registered, seeds reported, and reproducible on a single free-tier GPU.