Multilingual multimodal embedding models improve visual document search
pplx-embed-v2-late Technical Report
Information RetrievalComputation and LanguageMachine Learning
Summary
Finding information in text and images from many languages can be tricky. The authors designed two computer models that understand languages and images better than similar-sized models out there. They teach smaller, faster models by learning from a very large one, so the smaller models work well together. This helps computers find documents with pictures and text more accurately and quickly even when using different languages. Both models are made publicly available for anyone to use.
What this means in practice
- •For search engine engineers: Build improved search systems that retrieve documents with images and text from multiple languages using smaller, faster models alongside larger ones.
- •For digital archive maintainers: Index large collections of visual documents with a high-capacity model and respond to queries quickly with a compact model sharing the same embedding space.
Authors
Markus Krimmel, Louis Milliken, Ekaterina Glazkova, Maximilian Schall, Antoine Chaffin, Sedigheh Eslami, Jonas Golde, Denis Bykov, Claire Beamer, Jeremy Yang, Bo Wang
Abstract
In this report, we introduce pplx-embed-v2-late, a pair of 0.6B and 9B multilingual and multimodal late-interaction models. Both models achieve state-of-the-art results within their respective size classes across a range of text, image, and visual document retrieval benchmarks. On visual document retrieval tasks, pplx-embed-v2-late-0.6b is competitive with models several times larger than itself. The models are trained via representation distillation from an 18B teacher. Consequently, they share an embedding space that supports asymmetric deployment: documents can be indexed with the 9B model and queried with the low-latency 0.6B model. Both models are available at https://huggingface.co/collections/perplexity-ai/pplx-embed-v2.