Multi-vector visual document search sped up with smaller query models
ColNanoVDR: Document-Free Query Distillation for Multi-Vector Visual Document Retrieval via Optimal Transport
Information RetrievalComputation and LanguageComputer Vision and Pattern Recognition
Summary
Searching large collections of visual documents is slow because the computer runs a huge model every time. The authors developed a way to teach a smaller, faster model to ask the big model for help without needing to look at all the documents. They use a math technique called optimal transport to match parts of the smaller model’s questions to the big model’s answers. This method nearly keeps the search accuracy while making query processing up to 26 times faster.
What this means in practice
- •For enterprise search teams: Speed up visual document search queries without losing accuracy by using compact models trained with this distillation method.
- •For digital archive managers: Reduce storage and computation costs for large visual document collections by avoiding page encoding during query training.
Authors
Zhuchenyang Liu, Ziyi Wang, Yao Zhang, Yu Xiao
Abstract
Multi-vector retrievers built on vision-language models lead visual document retrieval (VDR), but they run a multi-billion-parameter query encoder on every search. Distilling this encoder into a small student that queries the teacher's existing index would remove the bottleneck. The standard recipe, however, matches the teacher's MaxSim scores and so requires encoding and caching every training page, which can reach terabytes of page tokens. NanoVDR avoids pages entirely by training on the teacher's query embeddings alone, but only for single-vector retrievers. We present ColNanoVDR, to our knowledge the first framework to bring this document-free distillation to multi-vector VDR. Its objective, OTW (Optimal Transport with Learned Weights), aligns the student's query tokens with the teacher's by entropic optimal transport, with a learned weight for each student token, and needs no correspondence between the two tokenizations. We prove that the resulting alignment cost bounds the MaxSim score difference on every page. Distilled from five state-of-the-art teachers, the 149M text-only students retain about 95% of their teachers' NDCG@5 on ViDoRe v1-v3 while encoding queries up to 26x faster. Under identical training, OTW matches score distillation while encoding no page and reading 12.6x less cached teacher data.