Joint subspace sparse attention speeds up long context language decoding
Joint Subspace Sparse Attention for Efficient Long-Context Decoding
Computation and Language
Summary
Reading and generating long texts with large language models is slow because they need to consider many past words at once. The authors designed a method called Joint Subspace Sparse Attention that smartly picks which past words to focus on, making the process faster without losing much accuracy. They create a shared low-dimensional space to better rank word relevance and use a new way to quickly gather needed information. Their approach works well on different model types and achieves big speed improvements on very long texts, even at huge model sizes.
What this means in practice
- •For machine learning engineers: Increase speed of long-sequence text generation by efficiently selecting context tokens during decoding with minimal accuracy loss.
- •For cloud ai service operators: Reduce computational demands and increase throughput for large language models handling very long inputs without retraining models.
Authors
Jiwon Song, Yoongon Kim, Jae-Joon Kim
Abstract
Sparse attention reduces the KV-cache traffic that dominates long-context LLM decoding, but practical methods largely operate at coarse granularity. Token-level selection offers finer control, yet presents two challenges: low-dimensional scoring may lose query--key structure needed for accurate ranking, while gathering scattered selected tokens can offset the resulting latency benefit. We introduce Joint Subspace Sparse Attention (JSSA) to address both challenges. For selection quality, JSSA learns a shared low-dimensional query--key subspace by minimizing token-score preservation error. We upper-bound the resulting non-convex objective with a tractable marginal objective that admits a closed-form spectral solution; direct refinement of the exact objective provides no measurable improvement in the tested settings. For execution efficiency, we develop a gather-free token-level indexed-attention kernel that supports distinct per-KV-head selections under the standard grouped-query attention (GQA) KV-cache layout. Across standard GQA, hybrid sliding-window/global attention, and Multi-head Latent Attention (MLA) models, JSSA consistently remains close to full-KV accuracy while avoiding the substantial generation-length inflation and architecture-dependent failures observed with competing methods. On an NVIDIA B200 GPU, JSSA achieves up to $6.43\times$ the decoding throughput of full-KV attention at 128K context. JSSA achieves reasoning accuracy comparable to model-native DSA without model training and reaches 96--99\% of its decoding throughput even at 700B scale large models.