Papers for

software developers in retrieval systems

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Content based addressing improves attention over long text sequences

Content-Based Addressing for Long Context

Abstract: Rotary position embedding (RoPE) uses each token's integer position to determine the rotation applied inside attention. This works well for local token order, but increasing context length creates a positional train-test mismatch: RoPE produces relative rotations at offsets not seen during training. Methods that rescale, interpolate, randomize, or bias positions specify how attention handles those offsets, but still derive positional information from a growing token counter. We instead divide a token stream into units, retain ordinary RoPE positions within each unit, and assign every completed unit an address computed from its content. Adding units then applies the same learned map to new content rather than extending a positional range or an identifier table. We prove that this construction preserves local RoPE exactly, leaves the attention comparison between two fixed tokens unchanged when other units are inserted or reordered, and does not create new relative rotations merely because more units are added. In a character-level Tiny Shakespeare diagnostic, a model trained on 256-character contexts has validation perplexity 4.04 at 256 characters and 3.82 at 4096, while continuous RoPE changes from 4.71 to 12.09. A second diagnostic shows that content-based addressing can retrieve and use information from multiple serialized facts. These are controlled shallow experiments, not scale benchmarks, but they support a direct prescription: use position to address locally and content to address across units.

Mon 7 SeptComputation and LanguageMachine Learning
The gist
Standard methods for helping AI models understand the order of words get confused when reading really long texts. The authors propose a way to split text into smaller parts and use the content of each part to help the model remember and relate information without getting mixed up by position numbers. This keeps the detailed order within small parts intact and improves the model’s understanding across the whole text. Tests on small text datasets showed their method works much better for long texts than traditional position-based methods.
Open → 2609.07314v1