RGSQ improves low-bit quantization for vision language models
RGSQ: Riemannian Geometry-Sensitive Quantization for Large Vision-Language Models
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Large vision-language AI systems can be hard to run efficiently because they need a lot of memory and time. Most methods to shrink these models work well for language-only systems but struggle with models that combine images and text. The authors introduced RGSQ, which better identifies parts of the model sensitive to simplification by using advanced geometry. This helps keep model accuracy high even when the model is compressed to use very little memory.
What this means in practice
- •For mobile device developers: Deploy vision-language AI models that use much less memory and run faster without large accuracy loss on phones and tablets.
- •For cloud ai platform engineers: Optimize the hosting of multimodal AI services by reducing hardware and energy costs through more efficient low-bit quantization.
Authors
Zhiping Wu, Dongdong Ren, Yangchengyu Zhou, Zhengjie Zhang, Wenbin Li, Hongbing Pan, Yang Gao
Abstract
Large vision-language models (VLMs) can be efficiently deployed under stringent memory and latency constraints through post training quantization (PTQ). However, most PTQ methods are designed for unimodal large language models (LLMs). These methods treat quantization errors as isotropic perturbations under the Euclidean assumption, which provides weak guidance on directions most sensitive to quantization in VLMs. Consequently, directly adapting unimodal PTQ approaches or solely employing modality-specific scaling often leads to uneven bit-width distribution and inconsistent performance in low-bit settings. To address these challenges, we propose Riemannian Geometry-Sensitive Quantization (RGSQ), which formulates quantization as a reconstruction problem under a unified Fisher-Riemannian metric. RGSQ identifies modality-specific sensitive directions via Riemannian manifold mappings built from modality-partitioned empirical Fisher factors and fused into a modality-aware Kronecker-structured metric. We then apply geometry-aligned rotations to reorient the local tangent frame, steering low-bit perturbations toward loss-insensitive axes. Finally, we apply a whitening transformation that maps the Riemannian objective to an equivalent Euclidean form, enabling standard unimodal PTQ methods to evaluate multimodal quantization error under their original assumptions. Across an extensive and diverse set of mainstream VLM benchmarks, RGSQ achieves the highest accuracy and stability under extremely low-bit settings (W2A8 and W3A8). It outperforms VLM-aware baselines, such as MBQ and MQuant, by up to 5.9% and surpasses single-modality improvements by up to 8.6%.