Efficient video search improves matching with compact reusable features
VEDJE: Video-Efficient Discriminative Joint Encoder for Scalable Video-Text Retrieval
Computer Vision and Pattern Recognition
Summary
Finding videos that match a text description can be tricky when scenes look similar but show different events. The authors introduce VEDJE, a method that compresses video features but keeps distinct information separate so it can be reused for multiple queries. This approach reduces the work needed to compare videos and text without losing accuracy. Tests on several video-text datasets show that VEDJE improves retrieval accuracy while storing much smaller video representations.
What this means in practice
- •For video platform engineers: Make video search faster and more accurate by storing compact, reusable video features that support many text queries without reprocessing videos each time.
- •For digital media archiving teams: Enable scalable retrieval from large video collections by compressing and caching key frame features without losing distinction between similar scenes.
Authors
Shahaf Wagner, Gabriele Serussi, Dan Ben Ami, Tomer Galanti, Chaim Baskin
Abstract
Finding the right video often requires distinguishing similar scenes in which different events occur. Joint matching improves retrieval, but processing rich video representations for each query is costly. VEDJE compresses features within sampled frames while keeping their representations separate in a reusable cache. Feature-change prediction supplies an auxiliary training signal that improves retrieval from the compressed cache without adding work at query time. On MSR-VTT, MSVD, DiDeMo, and ActivityNet, VEDJE improves R@1 over matched first-stage retrievers in both retrieval directions. On MSR-VTT, it reaches 59.8 text-to-video R@1 with a fine-tuned VideoCLIP-XL first stage. In the VideoPrism configuration, shrinking the per-video cache fourfold to 12 KiB preserves text-to-video recall within 0.2 points. These results show that accurate video search can operate on compact evidence, encoded once and reused as new queries arrive.