Apexquant improves low-bit model size without calibration data
ApexQuant: Data-Free Elastic Quantization by Residual Re-Isotropization
Machine Learning
Summary
Storing computer models in fewer bits can make them faster and smaller but often reduces their accuracy, especially without training data. The authors present ApexQuant, a way to compress these models step-by-step without needing any example data. It cleverly rotates and re-quantizes the leftover errors to keep reducing mistakes while making the model use less memory. This method works well for large language models and sensitive image data where training examples might not be available.
What this means in practice
- •For mobile app developers: Shrink large language models efficiently without needing calibration data for better on-device performance at low bit rates.
- •For medical data engineers: Compress clinical imaging models where patient data cannot be used for retraining or calibration, enabling privacy-compliant deployments.
Authors
Aksel Fristrup, Sumit Pandey, Ankit Kariryaa
Abstract
We introduce ApexQuant, a calibration-free quantization method that recursively re-quantizes the residual error, serving as a refinement layer on top of existing quantizers. We establish that a fresh random rotation returns each residual to the uniform distribution on the hypersphere, which characterizes the rate of progressive error decay across successive passes. This result lets us determine, before any weight is read, how many passes a layer needs for a target weight-space error. Every prefix is itself a valid lower-rate model, so one artifact serves several precisions. We instantiate ApexQuant with three interchangeable stages, scalar, $E_8$ and trellis, and validate it on four open-weight LLMs and on Earth-observation and medical domains where in-distribution data is often unattainable as imagery arrives under restrictive licences or due to patient material under privacy constraints. Progressive re-isotropization comes within a few percent of full precision at four bits and gives the best two-bit arm we measure, in a completely data-free setting.