Attention sinks remain at million token context despite new fixes
Do New Attention Mechanisms Actually Fix Attention Sinks at Million-Token Context?
Computation and LanguageArtificial Intelligence
Summary
Language models that read extremely long texts can struggle because they pay too much attention to the very first word, a problem called the attention sink. Some recent ideas tried to fix this by changing how the model focuses, but the authors found these fixes do not work as well when the model reads a million words at once. Instead, the cause seems to be the way the model is trained, not the model design itself. The authors created new tests to measure these issues and shared their findings and tools with others.
language modelsattention mechanismattention sinkcontext windowgated attentiontokentraining objectiveposition biasactivationrecency effect
Authors
Sara Rizwan, Samaanah Abdus Salam
Abstract
Long context language models now advertise windows of one million tokens, but two habits limit how much of that window is used. Attention heads with nothing useful to read still spend their budget on the first token, which is called the attention sink, and where a fact sits in the context changes whether the model finds it. Gated attention cut first token attention from 46.7 percent to 4.8 percent at NeurIPS 2025, and Kimi K3 pairs that idea with Kimi Delta Attention and Attention Residuals behind a one million token window, eight times past the range where these diagnostics have been reported. This paper asks whether the fix survives that jump. We build SinkProbe, a suite that measures sink mass, massive activation, position resolved recall and the recency gap, and apply it to four small models that differ only in how they mix tokens and depth. Three results follow. The training objective produces the sink, not the architecture. Gating did not reproduce its published effect at our scale. Sink mass, activations and position bias moved independently. Code, data and the measurement protocol are released at https://github.com/sararizwan7/Attention-Mechanisms-in-1M-Context-Window