Activation sparsity and kv-cache sparsity improve llm decoding speed differently
Where Activation Sparsity and KV-Cache Sparsity Cross in LLM Decoding
Machine LearningComputation and LanguagePerformance
Summary
When large language models generate text, they spend time reading two main types of data: model weights and stored past information called the key-value (KV) cache. The authors study two ways to make this faster by skipping some data—activation sparsity and KV-cache sparsity—and find that each works better depending on how much previous text is used. They provide formulas to predict when one method will outperform the other and confirm these with timing tests on real GPUs and long text sequences. Combining both methods leads to faster decoding than using either alone, and they share code so others can try it too.
What this means in practice
- •For ai engineers: Optimize language model decoding by choosing between activation sparsity and KV-cache sparsity according to context length to maximize speed.
- •For cloud infrastructure teams: Improve GPU resource efficiency during large language model inference by applying informed sparsity techniques matching workload context sizes.
Authors
Jungseob Lee, Seungyoon Lee, Seongtae Hong, Sugyeong Eo, Heuiseok Lim
Abstract
At each step, decoding one sequence with a large language model rereads the projection weights, whose traffic is fixed, and the key-value (KV) cache, whose traffic grows with context. Activation sparsity trims the first term and KV-cache sparsity the second, yet their reported speedups are hard to compare because each depends on context length and on the dense attention kernel it is measured against. We derive a byte crossover, the context length at which the two savings are equal, together with ideal speedup bounds for each branch and for their composition, from model dimensions and keep ratios alone. We then time both branches and their composition from 2K to 128K tokens on two GPUs after a dense prefill of real text, with dense and sparse modes reading the cache through the same split-K attention kernel. The projection branch leads at short context and the KV branch at long context, with speedups that follow their byte bounds up to fixed kernel costs. Adding these costs, measured in separate sweeps, lets the byte account predict the measured crossings of three keep-ratio pairs, a second model, and a second GPU to within 4.1K tokens. Timing the dense baseline with masked instead of split-K attention inflates the apparent speedup of the same KV policy about fivefold. An attention-scored KV selection answers the same passkey and multi-key placements as dense decoding up to 127K tokens, whereas a KV window misses most of them. Under matched perplexity budgets, activation sparsity composed with this selection decodes 14 to 26% faster than the best single branch on both GPUs. Code is available at https://github.com/js-lee-AI/ByteCross.