Training objective not architecture drives attention sink in long context models
Do New Attention Mechanisms Actually Fix Attention Sinks at Million-Token Context?
Computation and LanguageArtificial Intelligence
Summary
Some language models look at a huge amount of text at once, but they often get stuck paying too much attention to the first word, which limits how well they understand the rest. The authors studied different ways to fix this problem and found that changing the model’s goal during training, rather than its design, causes this issue. Techniques that seemed to reduce this problem at smaller scales did not work at the largest scales. This means that the behavior depends more on how models learn than on how they are built.
What this means in practice
- •For language model engineers: Design training processes that address attention sinks to improve performance on million-token inputs.
- •For natural language processing teams: Use SinkProbe to analyze how attention is distributed in very long text contexts for better model diagnostics.
Authors
Sara Rizwan, Samaanah Abdus Salam
Abstract
Long context language models now advertise windows of one million tokens, but two habits limit how much of that window is used. Attention heads with nothing useful to read still spend their budget on the first token, which is called the attention sink, and where a fact sits in the context changes whether the model finds it. Gated attention cut first token attention from 46.7 percent to 4.8 percent at NeurIPS 2025, and Kimi K3 pairs that idea with Kimi Delta Attention and Attention Residuals behind a one million token window, eight times past the range where these diagnostics have been reported. This paper asks whether the fix survives that jump. We build SinkProbe, a suite that measures sink mass, massive activation, position resolved recall and the recency gap, and apply it to four small models that differ only in how they mix tokens and depth. Three results follow. The training objective produces the sink, not the architecture. Gating did not reproduce its published effect at our scale. Sink mass, activations and position bias moved independently. Code, data and the measurement protocol are released at https://github.com/sararizwan7/Attention-Mechanisms-in-1M-Context-Window