MicroQonv reduces memory and speeds up neural network quantization
MicroQonv: Reshaping Convolution Tensors for Efficient Microscaling in Training and Inference
Hardware ArchitectureArtificial Intelligence
Summary
Quantizing neural networks means using fewer bits to store their numbers, which saves memory and speeds up calculations. But doing this efficiently in convolutional layers is tricky because of extra steps that increase memory use. The authors propose MicroQonv, a new method that reshapes data and quantizes it fewer times, cutting memory movement and storage needs by a large margin. This method keeps the accuracy high and helps with tasks like object detection and continual learning on small devices.
What this means in practice
- •For mobile ai developers: Enable more efficient memory use and faster inference in convolutional neural networks on edge devices.
- •For embedded systems engineers: Support low-bit quantization strategies in continual learning setups for edge AI applications with improved accuracy.
Authors
Romain Facq, Sami Ben Ali, Olivier Sentieys
Abstract
Microscaling quantization techniques are increasingly used to represent neural network parameters with 8 bits or fewer while preserving near-full precision accuracy. However, applying these methods efficiently in convolutional layers is not straightforward. A naive approach transfers full-precision weights and activations to processing units and quantizes each tensor twice, resulting in much more memory movement than expected. Additional overhead comes from the activation tensors, whose sizes grow substantially because of the im2col transformation applied before quantization. We propose MicroQonv, a way to combine microscaling with convolutional layers' forward and backward operations by quantizing each tensor only once and quantizing the activation tensor before applying a modified version of im2col: channel-batch-first im2col. MicroQonv reduces the quantization cost by a factor of $\times2$ for weights and gradients, and by up to $\times9$ for activations, at a negligible accuracy cost. It reduces memory movement and storage by up to $\times7.53$ compared to their full-precision counterparts. This way, MicroQonv reduces microscaling-quantized activation memory movement by $\times3.5$ for state-of-the-art object detection models YOLOV8nano and $\times2.2$ for YOLOV26nano. It also enables 4-bit microscaling in a quantized latent replay strategy for continual learning at the edge, improving accuracy by +5.7% to +11%.