Robust mixed-precision quantization speeds up vision models on edge CPUs
RAMP: Robust Adaptive Mixed-Precision Quantization for Edge CPU Vision Models
Computer Vision and Pattern RecognitionMachine LearningPerformance
Summary
Running smart computer vision programs on small devices like phones or sensors is hard because these devices have limited computing power. The authors studied many ways to safely simplify these programs to make them run faster without losing much accuracy. They found that a specific method called Jensen-Shannon Divergence works best to decide which parts of the program can be simplified. Also, grouping layers by how sensitive they are leads to faster models without hurting performance. Their work helps engineers make faster and more reliable vision apps on devices with limited power.
What this means in practice
- •For edge device developers: Optimize computer vision models for faster, reliable execution on low-power CPUs without GPUs using adaptive quantization policies.
- •For mobile app developers: Implement vision features in mobile apps with improved speed by selectively quantizing neural network layers based on sensitivity.
Authors
David Población-Criado, Dario Garcia-Gasulla, Eduardo Quinones
Abstract
Deploying deep learning models on edge CPUs is bottlenecked by computational and memory constraints. Mixed-precision quantization promises to reduce inference latency while preserving accuracy. However, quantization affects different layer types in inconsistent ways, so identifying where accuracy loss is minimized and latency reduction is maximized is critical, as the effect accumulates over a full deployment into substantial savings or unacceptable task degradation. Such identification relies on sensitivity metrics, proxies that estimate layer-wise degradation without evaluating the task accuracy of every candidate policy. Nevertheless, widely used metrics fail systematically on modern architectures. We present a systematic empirical study of 13 sensitivity metrics for layer-wise INT8 quantization across four distinctly different neural networks, and validate the resulting policies on two ARM64 platforms. Gradient-based sensitivity methods fail on 4 out of 8 model-hardware configurations and weight-based statistics on 2. In contrast, the Jensen-Shannon Divergence achieves zero catastrophic failures, reliably isolating the layers that cannot be safely quantized. A sensitivity metric alone does not define a policy, and the fixed thresholds typically used for that step are fragile over the highly skewed distributions of modern architectures. We address this with K-Means clustering, achieving near-lossless accuracy and a mean speed-up of $1.81\times$ over the full-precision model. Finally, we reveal that excluding from quantization the layers whose speed-up is negligible, regardless of their sensitivity, can be counterproductive, as it induces computational graph fragmentation and disables operator fusion. Our results yield concrete allocation policies for practitioners and researchers deploying quantized vision models on heterogeneous edge CPUs, without GPU access or gradient computation.