Vision language models quantized efficiently for faster edge deployment
SPHQuant: Efficient extreme low bit weight quantization for Vision-Language Models
Computer Vision and Pattern Recognition
Summary
Vision-language models combine images and text to understand both together, but their size makes them hard to run on smaller devices. The authors developed SPHQuant, a smart way to shrink these models by focusing on the direction and size of weight pieces separately. This approach avoids problems that happen when using very few bits to store model weights, keeping accuracy high. It also runs faster during decoding, making it easier to use these models on devices like laptops or phones.
What this means in practice
- •For mobile app developers: Deploy vision-language models on mobile devices with much lower memory use and faster response times than before.
- •For embedded systems engineers: Integrate multimodal AI capabilities into edge devices with limited compute power by using efficient low-bit quantized models.
Authors
Kewei Zhang, Zheng Chen, Haotong Qin, Yulun Zhang
Abstract
Recent foundation models are moving toward native multimodal Vision-Language Models (VLMs), making VLMs a central form of next-generation foundation models. However, their large language backbones make edge deployment difficult due to high memory footprint and memory-bound autoregressive decoding. Weight-only post-training quantization is a practical solution, but pushing VLMs to extreme low bit-widths remains challenging: existing rotation-free methods suffer from outliers at 2-3 bits, while rotation-based methods improve accuracy at the cost of additional runtime overhead. We propose SPHQuant, a rotation-free spherical weight-only quantization framework for VLMs. Instead of quantizing weights directly in Cartesian coordinates, SPHQuant decomposes each 8D weight vector into coordinate signs, radius, and a positive unit direction. This representation isolates outlier magnitude into the radius while keeping directions bounded and statistically regular. Based on this insight, SPHQuant allocates extra precision to the radius to mitigate accuracy degradation induced by outliers. It further uses a compact positive-direction codebook and fine-tunes codebook entries through angular parameterization to preserve the unit-sphere constraint. We also design a hardware-friendly GEMV kernel that keeps the direction codebook small enough for shared-memory lookup and packs radial bits efficiently. Experiments show that SPHQuant matches the performance of state-of-the-art extreme low-bit quantization methods while improving decode throughput over QTIP by 30.3% on RTX A6000. Code will be released in https://github.com/Pushazf/SPHQuant.