Progressive bit-depth training improves low-precision model compression
Progressive Multi-Ancestor Bit-Depth Distillation
Machine Learning
Summary
Small devices like phones or sensors often run into problems because their artificial intelligence models use too much memory and computing power. The authors found that directly shrinking these models to very small, low-precision forms makes training unstable and lowers accuracy. They introduced a new technique called PMABD, which slowly compresses the model in stages, using a group of less-compressed teacher models to guide the smaller, low-precision model. This step-by-step teaching helps keep the model accurate even when it's very small and efficient.
What this means in practice
- •For mobile app developers: Create smaller, faster AI models that run efficiently on smartphones with limited memory and processing power.
- •For embedded system engineers: Develop compressed neural networks that maintain accuracy on devices with tight energy and computational constraints.
Authors
Adil Mubashir Chaudhry, Osama Ahmad, Zubair Khalid, Murtaza Taj
Abstract
Model compression strategies are widely employed to reduce memory footprint and network complexity, particularly for devices with constrained computational, memory, and energy resources. Prior works that rely on simultaneous conversion from floating-point high-precision (FP32) to integer low-precision (INT4) representations and distillation into smaller models suffer from unstable training and drastic degradation of prediction performance. To address these limitations, we propose a unified framework, known as \textbf{P}rogressive \textbf{M}ulti-\textbf{A}ncestor \textbf{B}it-depth \textbf{D}istillation (PMABD), that progressively compresses the network while transferring knowledge through a growing pool of higher-precision ancestor teachers. PMABD generates a sequence of intermediate teachers that each learn from all higher-precision ancestors and jointly supervise the final target student. This multi-ancestor, multi-stage design stabilizes ultra-low-bit quantization by lowering quantization noise profiles across training and ensuring stable quantization. Experiments on CIFAR-10/100 with ResNet-20/32/18, and Tiny-ImageNet with MobileNetV2 show that PMABD outperforms state-of-the-art compression frameworks, results in 1.06$\%$ increase in performance of W2A2 (ResNet-18/CIFAR-100) student model. We show that a saturation-based stopping criterion contributes to improve the performance of our final student.