Quantization harms retrieval results more than classification accuracy shows
The Undetected Damage of Quantization on Retrieval and How to Fix It
Machine LearningArtificial Intelligence
Summary
When a model is simplified using quantization, its ability to classify things correctly may seem unchanged, but its performance in finding the top search result often changes a lot. The authors explain this by looking at how close the top two scoring items are in retrieval tasks, which isn’t the same in classification tasks. They find that if the score difference is large enough compared to rounding errors, quantization doesn’t change the top choice. This insight allows for smarter use of extra precision where it matters to keep results reliable.
What this means in practice
- •For machine learning engineers: Identify which parts of a retrieval model need higher precision to maintain correct top search results after quantization.
- •For data platform teams: Route uncertain classification inputs to full precision models to preserve accuracy while saving computation.
Authors
Luca Zhou, Alessandro Zirilli, Daniele Solombrino, Roberto Dessì, Emanuele Rodolà
Abstract
We show that a quantized model that keeps its classification accuracy still changes $14$ to $46\%$ of its top-1 retrieval results, and that aggregate ranking metrics reveal only part of this damage. We tie this failure to the gap between the two highest scores and use that gap to decide when a quantized answer can be trusted and where additional precision should be spent. We show that the top-1 result is guaranteed to survive quantization only when this gap exceeds twice the largest rounding error. In classification, scores are the logits, and the loss function pushes the correct class away from other classes, encouraging this gap. In retrieval, scores are query-document scores, and nothing separates the top-1 item from the second. This gap can be measured without labels. Before deployment, it predicts which models will break under quantization, and at deployment time it tells, per input, whether the quantized answer still matches the full-precision answer. Most classification inputs have a gap wide enough to trust the quantized answer, but few retrieval queries do. That gap motivates a different fix in each task. In retrieval, spending extra bit-width on the layers whose quantization moves the gap most recovers up to three-quarters of an extra bit's benefit for half its cost. In classification, routing the few low-gap inputs to full precision recovers most of the lost accuracy at a fraction of the cost.