FoldAttention speeds up AI decoding with deterministic exact sums
FoldAttention: Declared-Reference Softmax for Fast Decode and Deterministic Backward
Machine LearningDistributed, Parallel, and Cluster Computing
Summary
When generating text or other sequences step-by-step, computers spend a lot of time calculating attention with huge data stored in memory. The authors introduce FoldAttention, a new way to compute attention that fixes a reference point before reading data, making each weight final when calculated. This lets the system skip unnecessary data early and combine partial results exactly, speeding up the process and making it deterministic. Their approach runs faster than previous methods on modern GPUs without losing accuracy.
What this means in practice
- •For machine learning engineers: Accelerate transformer decoding on GPUs by reducing attention calculation load and enabling deterministic backward passes without slower runtimes.
- •For gpu kernel developers: Design faster and exact attention kernels that gate memory reads and combine partial sums efficiently for complex transformer models.
Authors
Sriman Achanta
Abstract
Autoregressive decode repeatedly streams a growing KV cache, making attention a major cost at long context. Existing high-performance kernels use online softmax, which discovers a row's normalization reference as it scans keys. Earlier contributions therefore remain provisional and may require rescaling. We argue that the reference need not be discovered: softmax is invariant to a common shift, so the reference only has to keep the weights in range. We present FoldAttention, an additive formulation of softmax attention that fixes a finite reference $Z_i$ before scanning the KV cache. Each weight $2^{s_{ij}-Z_i}$ is then final when computed, so contributions add across disjoint key ranges and their quotient equals softmax attention in real arithmetic. We use this property to develop two techniques for Hopper decode: (1) final weights gate key and value reads before the bytes are fetched, and a per-call depth $T$ cuts keys below $2^{-T}$ while keeping their mass, and (2) additive partials compose split KV and shared-prefix cascades without rescaling. On H100 at $T=16$, FoldAttention decodes seven real-model generations 1.36-2.30$\times$ faster than the fastest BF16 baseline, and up to 3.09$\times$ faster across MHA and GQA shapes, at an error within 1.5% of the lowest BF16 error on six of the seven; reading every key, it is 1.14-1.30$\times$ faster at matched error. We validate on Qwen3-8B that a whole decode step is up to 1.46$\times$ faster while likelihood and long-context accuracy match those under BF16 kernels. The same principle makes the backward deterministic: CTAs round bounded partial gradients onto an integer grid declared before the reduction and add them in any order. FoldAttention thereby removes the determinism tax: its deterministic backward is up to 1.84$\times$ faster than deterministic FlashAttention-3/4 and 1.05$\times$ faster than the fastest nondeterministic kernel.