SVD Contextual Sparsity Predictors for Fast LLM Inference
Machine Learning
Summary
The authors developed a way to make large language models run faster while keeping their accuracy mostly intact. They focus on a part of the model called the feed-forward network and use a math technique called singular value decomposition to predict which parts can be skipped during processing. Their method doesn’t require extra training and works well on special hardware like CUDA devices. Tests showed their approach speeds up the model by almost twice with very little loss in performance for tasks like math and coding. This helps in using such models on smaller devices.
Authors
Georgii Serbin, Kirill Koshkin, Zhongao Sun, Anastasiya Bistrigova, C. C. Korikov
Abstract
Contextual sparsity is one of the approaches used to reduce computational complexity in the inference process of large language models (LLMs). Existing techniques for efficient LLM inference acceleration based on contextual sparsity with minimal accuracy degradation require training sparse pattern predictors. This paper presents a framework for accelerating inference of ReGLU-based feed-forward networks (FFNs) within LLMs. The proposed framework provides a fast, training-free method for building sparse pattern predictors using truncation-aware singular value decomposition (SVD) of the gate projection matrix, along with a threshold calibration algorithm, and inference executors supporting conditional computation on CUDA and CANN devices. Experiments on three sparse LLMs with an average activation sparsity level of 90% in the FFNs demonstrate up to a 1.8x reduction in end-to-end decoding time while maintaining less than 1% degradation in benchmark scores on tasks involving complex math and code generation. This work advances the deployment of LLMs on edge devices.