Adaptive sparsity method improves large language model efficiency
TopK-Guided: Adaptive, Budget-Aware Activation Sparsity for Efficient LLM Inference
Artificial Intelligence
Summary
Large language models can run faster by ignoring less important parts of their calculations. Existing methods either change how much they ignore for each word but can’t control how much is skipped, or fix a skipping amount that applies to every word and part of the model the same way. The authors introduce a new approach that combines careful control over how much each word skips with adjustments based on how sensitive each part of the model is to skipping. Their method improves accuracy and speeds up processing, especially when skipping a lot of calculations.
What this means in practice
- •For ai infrastructure engineers: Reduce compute costs for large language model serving by implementing adaptive sparsity that balances accuracy and efficiency per token and model block.
- •For mobile app developers: Enable faster on-device language model inference by applying budget-aware sparsity techniques to maintain quality while using fewer resources.
Authors
Mukund Agarwalla, Chih-Jen Lin
Abstract
Activation sparsity speeds up large language model (LLM) inference by setting unimportant activations to zero so that the corresponding computations can be skipped. Existing training-free methods, however, make different trade-offs: threshold-based methods such as TEAL adapt the sparsity level to each token but do not tightly control the realised sparsity, while TopK-based methods such as WINA enforce a fixed sparsity level but use the same sparsity budget for every token. Both also apply the same budget across transformer blocks, despite large differences in block sensitivity. We introduce TopK-Guided, a training-free method that addresses both limitations by combining bounded token-level sparsity adaptation with sensitivity-aware block-level budget allocation. Across Llama-2 and Llama-3 models, TopK-Guided consistently improves perplexity and downstream accuracy over TEAL and WINA while preserving essentially the same sparsitydependent projection compute as WINA, with the largest gains at high sparsity. Ablations show that both components provide complementary improvements.