Content based addressing improves memory in long text models
Content-Based Addressing for Long Context
Computation and LanguageMachine Learning
Summary
Understanding where a word fits in a long story is tricky for computers because they often rely on counting words to remember order. The authors found a way to divide text into chunks and use the meaning of each chunk to help the computer keep track, rather than just its position in the whole text. This method keeps local word order intact while allowing the computer to handle much longer texts without confusion. Their tests showed that this idea helps the model understand much longer texts better than standard counting methods. It suggests that using content to find information across chunks is a better way to remember context over long passages.
rotary position embeddingattention mechanismpositional encodingcontext lengthcontent based addressinglanguage modelstokenizationperplexityrelative positioningnatural language processing
Authors
Mahesh Godavarti
Abstract
Rotary position embedding (RoPE) uses each token's integer position to determine the rotation applied inside attention. This works well for local token order, but increasing context length creates a positional train-test mismatch: RoPE produces relative rotations at offsets not seen during training. Methods that rescale, interpolate, randomize, or bias positions specify how attention handles those offsets, but still derive positional information from a growing token counter. We instead divide a token stream into units, retain ordinary RoPE positions within each unit, and assign every completed unit an address computed from its content. Adding units then applies the same learned map to new content rather than extending a positional range or an identifier table. We prove that this construction preserves local RoPE exactly, leaves the attention comparison between two fixed tokens unchanged when other units are inserted or reordered, and does not create new relative rotations merely because more units are added. In a character-level Tiny Shakespeare diagnostic, a model trained on 256-character contexts has validation perplexity 4.04 at 256 characters and 3.82 at 4096, while continuous RoPE changes from 4.71 to 12.09. A second diagnostic shows that content-based addressing can retrieve and use information from multiple serialized facts. These are controlled shallow experiments, not scale benchmarks, but they support a direct prescription: use position to address locally and content to address across units.