LongCat Sparse Attention: Taming the Lightning via Streaming-aware Hierarchical Cross-Layer Indexing
2026-08-03 • Artificial Intelligence
Artificial IntelligenceComputation and LanguageDistributed, Parallel, and Cluster ComputingMachine Learning
AI summaryⓘ
The authors identify efficiency problems in a method called DeepSeek Sparse Attention (DSA) used for handling very long texts. They propose LongCat Sparse Attention (LSA), which improves speed and hardware use by reorganizing data layout, sharing computations across layers, and gradually narrowing down important parts for each word. Their approach matches the performance of full attention methods while supporting extremely long contexts, up to one million tokens. They also provide an open-source version to help others explore this technology.
Sparse AttentionLong-Context ModelingIndexing OverheadHardware EfficiencyHierarchical IndexingCross-Layer DistillationHigh Bandwidth Memory (HBM)Transformer ModelsModel ScalingOpen Source
Authors
Wen Zan, Jiaqi Zhang, Jianchao Tan, Hong Liu, Cunguang Wang, Xiang Li, Duyue Ma, Guanyu Wu, Yifan Lu, Fengcun Li, Yerui Sun, Peng Pei, Yuchen Xie, Xunliang Cai
Abstract
DeepSeek Sparse Attention (DSA) enables efficient long-context modeling through its Lightning Indexer. However, practical deployment remains constrained by the indexer's expensive $O(L^2)$ scoring overhead and the hardware-inefficient, discontinuous memory-access patterns induced by its outputs. To address these system-level bottlenecks, we introduce LongCat Sparse Attention (LSA), a hardware-algorithm co-designed framework comprising three complementary and orthogonal strategies: (1) Streaming-Aware Indexing, which selectively converts scattered KV entries into hardware-aligned contiguous layouts to enable coalesced HBM access; (2) Cross-Layer Indexing, which amortizes indexing overhead by reusing the results produced by a single layer across consecutive layers, supported by cross-layer distillation; and (3) Hierarchical Indexing, which adopts a coarse-to-fine scoring scheme to progressively narrow the candidate set for each query, thereby substantially reducing indexing computation. Extensive scaling experiments, ranging from 69B-A3B to 560B-A27B models, demonstrate that LSA consistently achieves performance on par with full attention across both general-purpose and long-context benchmarks. Moreover, LSA supports native training with context lengths of up to one million tokens and underpins the development of LongCat-2.0 (1.6T-A48B). To facilitate further research, we also introduce and open-source LongCat-Flash-Lite-Sparse (69B-A3B), which integrates LSA into LongCat-Flash-Lite and incorporates an updated long-context training corpus.