AI summaryⓘ
The authors address how running large language models (LLMs) on edge devices faces challenges due to limited memory and speed, especially when using sparse and low-bit quantized data. They identify that metadata used to track sparsity can become a bottleneck even when weights are compressed. To fix this, they introduce a new ratio called Payload-to-Metadata Ratio (PMR) and propose UnionSparse, a method that efficiently encodes sparse data and speeds up matrix multiplication needed for inference. Their approach significantly outperforms existing methods in speed during low-bit sparse computations on GPUs. This work highlights the importance of managing metadata traffic in efficient LLM inference on edge devices.
Large Language Models (LLMs)Edge inferenceSparsityLow-bit quantizationPayload-to-Metadata Ratio (PMR)Sparse matrix multiplication (SpMM)Bitmap EncodingShared-Memory Parallel DecodingGPU accelerationMatrix multiplication kernels
Authors
Tianhao Jiang, Hang Gu, Teng Wang, Qianyu Cheng, ZhenDong Zheng, Cheng Tang, Qiyue Su, Wenqi Lou, Lei Gong, Chao Wang, Xi Li, Xuehai Zhou
Abstract
Edge LLM inference combines sparsity and low-bit quantization to meet device memory, latency, and power limits. Yet quantization shrinks weight payloads without proportionally reducing sparse metadata, so index traffic and nonzero extraction become critical SpMM bottlenecks. We introduce the Payload-to-Metadata Ratio (PMR) and show that improving PMR raises effective compute intensity in decoding. We present UnionSparse, an index-efficient framework that combines Index-Efficient Bitmap Encoding (IE-BME) with a SpMM kernel using Low-Bit Shared-Memory Parallel Decoding (LSPD). IE-BME amortizes metadata and aligns sparse traversal with fragment assembly, while LSPD improves small-batch execution. Under W4A4 quantization and 30%--70% sparsity, UnionSparse outperforms FlashLLM and SpInfer by 2.30x and 1.43x, and CUTLASS and cuBLAS Tensor Core by 1.56x and 3.46x, respectively. These results establish payload-extraction efficiency as a first-order concern for low-bit sparse inference on edge GPUs. Source code is available at: https://github.com/Victor-Alen/UnionSparse.