Long-context language models speed up prompt processing using radius-based sparse attention
RBS-Attention: Radius-Bounded Sparse Prefill for Long-Context Large Language Models
Artificial Intelligence
Summary
Large language models struggle to quickly process really long text prompts because they need to look at every word before they start generating answers. The authors found that picking groups of words by their average importance sometimes misses very relevant words hidden inside the group. They introduced a method called RBS-Attention that uses two ways to select important text blocks, one based on overall importance and another that detects blocks that might have very important hidden words. This approach speeds up the model’s processing a lot without losing much accuracy, enabling much longer contexts to be handled efficiently.
What this means in practice
- •For large language model engineers: Enable faster processing of extended prompts in large language models using radius-based sparse block selection to reduce prefill costs.
- •For video content analysis teams: Improve handling of long video-related text inputs by using efficient attention methods to speed up language model inference.
Authors
Chuxu Song, Jiuqi Wei, Zhencan Peng
Abstract
Long-context large language model inference is increasingly limited by prefill, where dense self-attention processes the entire prompt before generation begins. Sparse block selection can reduce this cost, but a block centroid may hide a highly relevant token among many irrelevant ones. We call this failure mode mean dilution and propose RBS-Attention, a training-free sparse-prefill method with two complementary selection branches. A centroid base branch captures average relevance, while a rescue branch uses the maximum key-block radius and its prompt-, layer-, and head-dependent distribution to identify blocks at risk of underestimation. Independently thresholding the two branches and combining their masks controls the contribution of rescue blocks while preserving regular block-sparse FlashAttention execution. On H100 GPUs, RBS-Attention achieves 20.65$\times$ standalone prefill-attention speedup, 11.92$\times$ vLLM prefill-attention speedup, and 5.97$\times$ end-to-end time-to-first-token speedup at 128K on Qwen3-30B-A3B-Instruct-2507-FP8. On the dense Qwen3-32B model, it obtains 88.65 overall RULER accuracy versus 89.52 for dense attention; LongBench-v2, InfiniteBench, and Video-MME provide additional quality evaluation. Supporting experiments measure actual retention, compare selectors at matched density, and characterize block-size, threshold, and memory behavior. Together, these results support radius-adaptive dual-branch selection as an effective approach to long-context prefill.