Channel-wise and token-aware quantization improves visual state models
Channel-Wise and Token-Aware Post-Training Quantization for Visual State Space Duality
Computer Vision and Pattern Recognition
Summary
Models that analyze images can be made faster and use less memory by reducing the precision of their calculations. The paper finds that the biggest challenge is accurately adjusting the way these models handle different parts of the data during this reduction. The authors designed a method that finely tunes how each part of the model scales data, keeping accuracy high even with less precision. Their approach works well on popular image and object recognition tasks and speeds up processing on advanced hardware.
What this means in practice
- •For machine learning engineers: Reduce precision in visual state space models for faster and memory-efficient inference without losing accuracy on standard image benchmarks.
- •For computer vision developers: Maintain strong object detection, instance segmentation, and semantic segmentation performance while enabling efficient low-precision execution.
Authors
Jonghyeon Lim, Changhoon Yim
Abstract
State space models (SSMs), particularly Mamba, have emerged as efficient alternatives to attention-based architectures and have been extended to vision through ViM, VMamba, and Visual State Space Duality (VSSD). Yet the low-bit post-training quantization (PTQ) behavior of VSSD remains insufficiently understood. A weight-activation split on VSSD-Tiny identifies activation quantization as the dominant low-bit bottleneck, while representative inputs to selected VSSD-backbone linear layers exhibit strong channel-wise magnitude variation and token-localized extremes. We propose the Channel-wise Token-balanced Output-Aware Clipping (CTOAC) method, which learns per-input-channel clipping bounds by minimizing a token-balanced reconstruction loss on the corresponding linear outputs. Only the selected linear layers and their input activations are quantized; other backbone operations retain their original precision. Across VSSD-Tiny, VSSD-Small, and VSSD-Base, the proposed CTOAC method retains ImageNet-1K accuracy and remains substantially more robust than the evaluated baselines at more aggressive precision settings. Applying the same quantization scope to VSSD backbones on COCO and ADE20K preserves strong object detection, instance segmentation, and semantic segmentation performance. An optimized RTX 4090 deployment configuration achieves up to 1.42x end-to-end speedup over FP32.