Weight-only quantization advice fails on text retrieval embedders
Where Post-Training Quantization Breaks Text Embedders: A Measured Map Across Four Embedder Families
Information RetrievalComputation and Language
Summary
Shrinking machine learning models by reducing their numerical precision is common, but the usual rules for doing this don’t work well for text retrieval models. The authors show that protecting the biggest part of the model isn’t always most effective, and that different parts of the model respond differently depending on the precision used. They also find that a smaller model trained specifically for a task can outperform a larger quantized model but only on that task. This study provides detailed measurements that challenge assumptions carried over from other kinds of models.
What this means in practice
- •For ml model compression engineers: Avoid relying on standard quantization heuristics when compressing text retrieval embedder models to better preserve performance at very low bit precisions.
- •For search engine developers: Deploy smaller distilled embedding models that perform better than large quantized models for specific tasks, saving memory while maintaining quality.
Authors
Hyojung Han
Abstract
Weight-only post-training quantization is the cheapest way to shrink a retrieval embedder, and the received advice for applying it -- protect the embedding table, allocate bits by module sensitivity, prefer a ranking-aware objective over weight reconstruction -- was carried into LLM quantization largely intact. We test that advice on retrieval embedders directly, quantizing five checkpoints from four architecture families across a grid of bit widths and group sizes, and isolating the embedding, attention and feed-forward blocks at each width. Every heuristic fails to transfer as stated. The embedding table never emerges as the dominant isolated protection priority in any family, despite being the largest tensor in several of them. Module sensitivity does not survive as a transferable ordering: at INT4/g16 the spread between modules is too small to allocate against, at INT3 the ordering becomes family-dependent and joint damage stops being the sum of its parts, and at INT2 comparable reconstruction error accompanies retention ranging from 1.3 to 65.9 percent of full precision. A cheap reconstruction proxy is useful for screening uniform bit widths but substantially less reliable for choosing which tensors to protect; its apparent strength across the whole grid is a range-extension artifact. A distilled 109M student at INT3 holds 78.04 NDCG@10 in 68.4 MB and dominates the extreme-PTQ arm of its own 0.6B teacher, 297.9 MB at 64.46, on both size and quality -- but only inside the task it was distilled for. Sizes are byte counts of files that exist rather than arithmetic estimates, and the measurement repository carries the byte provenance for every one of them.