Adding 2 4 sparsity is less effective than lowering precision for large language models
Is 2:4 Sparsity Still Worth It? Semi-Structured Sparsity versus Quantization in LLM Inference
Machine LearningPerformance
Summary
The paper explores two ways to make large language models run faster on NVIDIA GPUs: using 2:4 sparsity (dropping half of certain groups of data) and lowering the precision (using fewer bits for numbers). The authors found that lowering precision is usually better than adding 2:4 sparsity because dropping data in 2:4 sparsity doesn’t save as much energy or improve speed compared to reducing precision. However, when the lowest precision is already used, 2:4 sparsity can speed things up, but it usually needs retraining to keep the model accurate. The study tested these methods on different GPUs and layers of transformer models.
What this means in practice
- •For ai inference engineers: Choose quantization over 2:4 sparsity when reducing precision to improve speed and accuracy on NVIDIA GPUs in transformer model inference.
- •For gpu hardware designers: Design future GPUs with support for effective combinations of sparse patterns and low precision quantization to optimize transformer inference.
Authors
Xiuyi Qin
Abstract
NVIDIA tensor cores have supported 2:4 semi-structured sparsity since Ampere, and the pattern composes with quantization. When the two are combined, the practical question is which step to take next: from a given precision, add 2:4 sparsity or drop to the next lower precision? We compare the two at equal bits, with sparsity metadata counted, and at equal nominal tensor-core tiers, over all six GEMM operands of a transformer layer, with kernels measured on three GPU generations and online pruning and quantization included. Our results show that as long as a lower precision is available, adding 2:4 is the worse step, both less accurate and slower, because the energy that 2:4 discards from each group of four along the hardware-forced axis does not shrink with bits, while quantization error does. Once quantization reaches the lowest precision the hardware supports, 2:4 still speeds up linear layers by 1.3--1.7\x{}, and on an H100 a retrained 2:4 model served in FP8 by vLLM reaches 27\% higher prefill throughput than the dense FP8 model, with GSM8K accuracy on par. Its quality has to be won back by retraining, however, which restores perplexity long before reasoning; on Blackwell, whose sparse FP4 uses a pairwise 4:8 pattern, the loss is larger and our retraining budget did not recover it.