Simple linear corrections improve compressed document search quality
Simple Linear Corrections for Compression in Late Interaction
Information RetrievalComputation and LanguageMachine Learning
Summary
Searching large collections of documents using detailed vector data is often slow and costly. The authors studied ways to compress these vectors to save space and speed up searches, but this usually reduces accuracy. They found that adding a small, simple correction step after compression can bring back much of the lost quality without redoing the whole process. Their method works well across different models, data sets, and compression levels, making search systems more efficient.
What this means in practice
- •For search engine engineers: Reduce storage and scoring costs in multi-vector document search by applying lightweight linear corrections to compressed vectors to recover retrieval quality.
- •For information retrieval system architects: Maintain search accuracy in systems using aggressive document-vector compression without retraining encoders or recollecting embeddings by adding post-compression linear score adjustments.
Authors
Mihir Agarwal, Sabyasachi Basu, Viraj Agrawal
Abstract
Late-interaction (multi-vector) models represent documents using a set of token-level vectors, providing a middle ground between cross-encoders and dense encoders in terms of quality and cost of embedding. However, compared to the latter, the indices are costly to store and score. Pooling and pruning based compression techniques are deployed widely but deteriorate in quality noticeably with aggressive compression. We investigate whether a lightweight change to the scoring function can recover retrieval quality after document-vector compression. For each combination of encoder, compressor, and vector budget, we learn a global linear correction by distilling the uncompressed model's score distributions over candidate documents. The encoder and compressor remain frozen, and the correction can be applied to each query at search time or once, offline, to the stored vectors. We evaluate the method with ColBERTv2 and Jina v4 on MS MARCO and BEIR, and with ColPali2 on the ViDoRe v3 visual document retrieval benchmark. Experiments cover four training-free compression methods, multiple vector budgets, in-domain adaptation, and transfer from MS MARCO to BEIR. In the in-domain setting, the correction improves over baseline compression in 154 of 160 BEIR settings, while transfer from MS MARCO is more variable across models and datasets. We also find that matrices trained with synthetic queries can improve over baseline compression on ViDoRe v3. These results show that a small learned correction can recover some of the quality lost to compression without retraining the encoder or recompressing the documents.