Layerwise method speeds up and strengthens neural network quantization
Layerwise Error Attribution for Fast and Robust Mixed-Precision Post-Training Quantization
Machine Learning
Summary
Reducing the size of neural networks often involves assigning fewer bits to represent each layer’s data, which is tricky and slow to do well. The authors came up with a fast and reliable way to decide how many bits each layer should get, even if the example data used to guide this decision is messy or corrupted. Their approach breaks down the error caused by simplifying each layer and uses this to quickly pick bit assignments without heavy computations. Tests showed their method is much faster and can keep or improve the quality of tasks like image denoising, especially when the calibration data is imperfect.
What this means in practice
- •For mobile app developers: Compress neural networks in apps to run faster and use less memory while maintaining quality, even with imperfect calibration data.
- •For cloud service engineers: Rapidly optimize bit allocation in trained models to reduce cloud storage and compute costs, improving deployment efficiency.
Authors
Samy Houache, Yann Traonmilin, Jean-François Aujol
Abstract
Mixed-precision post-training quantization is a network compression method that assigns bits layer by layer, under a global memory budget using a small calibration set. The main difficulties are to overcome the combinatorial nature of the allocation problem and to manage the sensitivity to small, potentially corrupted databases. Hence, an efficient allocation method should be fast to compute and preserve model quality when calibration data are corrupted. To design such a method, we derive a layerwise probabilistic analysis of the quantization error that separates propagated error from the local perturbation introduced at a given layer. We use this local term to build a separable score for a simple allocation algorithm, that requires no external solver. The probabilistic nature of our approach brings robustness to corrupted data. On denoising tasks with DRUNet, with an average budget of 4 bits per weight, our method matches or improves state-of-the-art mixed-precision baselines under clean calibration, and is more robust to corrupted calibration, with PSNR gains of up to 7.5 dB under the tested corruptions. Experiments show bit-allocation speed-ups from 28x to 2,570x over the studied baselines. For quantized diffusion models, our experiments show that a direct application of our framework also improves the state-of-the-art.