Multi-vector visual document indices can leak readable pages
Inverting Multi-Vector Visual Document Indices
Information RetrievalComputation and LanguageComputer Vision and Pattern Recognition
Summary
Document search systems often store pages as many small image pieces called vectors, thought to be safe because people can't read the page from these alone. The authors show that by reversing this process, someone with access to these vectors can recreate much of the original page text, exposing sensitive content. They tested simple defenses like mixing up the vectors, which reduce the risk but don't stop all attacks. This means such document indexes need similar protections as the documents they represent.
What this means in practice
- •For cloud service providers: Protect multi-vector document indexes by applying vector shuffling to reduce the risk of sensitive page reconstruction from stored embeddings.
- •For enterprise cybersecurity teams: Apply this paper's methods to evaluate and mitigate leakage risks from storing sensitive document vectors in third-party services.
Authors
Zhuchenyang Liu, Yao Zhang, Yu Xiao
Abstract
Prevailing multi-vector visual document retrievers store each page as about a thousand patch vectors, often in vector databases run by a third party. Since no one can read a page from its vectors, this index is easily treated as less sensitive than the page. However, because the index keeps one vector per patch in raster order, and each vector is computed by a vision-language model pre-trained to read documents, we hypothesize that whoever runs or breaches the store can reproduce a page from its index alone. We frame inversion as conditional document image generation and infer from the vectors what the attack needs: the encoder, the page shape and, for shuffled vectors, their order. On the ViDoRe v3 benchmark, pages inverted from raw indices recover 47% of the words and 45% of the sensitive tokens. Used as queries against the stored indices, they rank their source page first 98.4% of the time. We test two cheap protections, token pooling and shuffling, which both cut word recall to about 8%. A model that restores the order of a shuffled index raises the share of source pages ranked first from 3.8% to 93.5%, while inverting a pooled index remains open. To test generalisation, we apply the same attack unchanged to another multi-vector retriever: its inverted pages still rank their source page first 70.2% of the time, though its word recall stays below a nearest-neighbour baseline. Multi-vector visual document retrievers are therefore vulnerable to inversion through their stored index, which should be protected like the documents it encodes.