Efficient quantization aware training improves vision language models on edge
Efficient Quantization-Aware Distillation with Cross-Modal Alignment for Edge Vision-Language Models
Computer Vision and Pattern Recognition
Summary
Running large vision-language AI models on small devices like phones is hard because these models use a lot of memory and computing power. The authors propose a new method that trains smaller versions of these models to work better within limited hardware by combining two training steps into one, making the training more consistent. They also improved how the model handles different types of image inputs by letting one type help guide another. Their approach makes running these models on edge devices more accurate and efficient.
What this means in practice
- •For mobile app developers: Enable efficient image and text classification on smartphones using smaller, optimized vision-language models.
- •For security camera manufacturers: Improve object recognition in low-power cameras by enhancing non-RGB image processing with quantized lightweight models.$Commercial implications: Supports sales of cost-effective cameras with AI capabilities for better security and monitoring.
Authors
Jinwoo Jeon, GyuYeop Do, Yubin Lim, Nam-Joon Kim, Hyun Gon Ryu, Hyuk-Jae Lee, Byung-Jun Lee
Abstract
Large-scale vision-language models (VLM) such as CLIP enable strong open-vocabulary reasoning, yet deploying these capabilities on resource-constrained edge devices remains challenging. EdgeVL addresses this problem by distilling CLIP representations into lightweight multi-modal encoders and applying quantization-aware training (QAT) for efficient Open-Vocabulary Classification (OVC) on edge hardware. However, its two-stage optimization applies different objectives for distillation and QAT, and contrastive learning is performed within the quantized student space, which can result in inconsistent optimization and reduced training efficiency. Moreover, identical supervision across RGB and non-RGB modalities may lead to modality imbalance. We propose a unified framework for quantized semantic distillation tailored to edge deployment. By jointly optimizing distillation and quantization within a unified teacher-anchored framework, our method ensures consistent training under quantization, suppressing hard negatives and enlarging decision margins. Additionally, we design a lightweight cross-attention adapter that enhances non-RGB representations through RGB-guided semantic transfer, narrowing the modality gap. Extensive experiments demonstrate consistent improvements on non-RGB modalities while maintaining deployment efficiency.