TriAttention: Efficient Long Reasoning with Trigonometric KV Compression
2026-04-06 • Computation and Language
Computation and LanguageComputer Vision and Pattern Recognition
AI summaryⓘ
The authors identify a memory bottleneck problem in large language models when they do long reasoning tasks. They find that current methods using recent attention scores are unstable due to how queries shift with positions. Instead, they look before this shift (pre-RoPE space) and notice queries and keys cluster around fixed points, which helps predict which keys are important. They propose TriAttention, a method that uses this clustering and trigonometric patterns to better estimate key importance, improving memory use and speed without losing accuracy. Their method works well even on a single consumer GPU where other methods fail due to memory limits.
Large Language ModelsKV CacheRoPE (Rotary Position Embedding)Attention MechanismQuery/Key VectorsPre-RoPE SpaceMemory BottleneckTriAttentionToken GenerationThroughput
Authors
Weian Mao, Xi Lin, Wei Huang, Yuxin Xie, Tianfu Fu, Bohan Zhuang, Song Han, Yukang Chen
Abstract
Extended reasoning in large language models (LLMs) creates severe KV cache memory bottlenecks. Leading KV cache compression methods estimate KV importance using attention scores from recent post-RoPE queries. However, queries rotate with position during RoPE, making representative queries very few, leading to poor top-key selection and unstable reasoning. To avoid this issue, we turn to the pre-RoPE space, where we observe that Q and K vectors are highly concentrated around fixed non-zero centers and remain stable across positions -- Q/K concentration. We show that this concentration causes queries to preferentially attend to keys at specific distances (e.g., nearest keys), with the centers determining which distances are preferred via a trigonometric series. Based on this, we propose TriAttention to estimate key importance by leveraging these centers. Via the trigonometric series, we use the distance preference characterized by these centers to score keys according to their positions, and also leverage Q/K norms as an additional signal for importance estimation. On AIME25 with 32K-token generation, TriAttention matches Full Attention reasoning accuracy while achieving 2.5x higher throughput or 10.7x KV memory reduction, whereas leading baselines achieve only about half the accuracy at the same efficiency. TriAttention enables OpenClaw deployment on a single consumer GPU, where long context would otherwise cause out-of-memory with Full Attention.