Foundation models improve video search by adapting processing depth
Beyond Similarity: Foundation Models as an Efficient Backbone for Training-Free Composed Video Retrieval
Computer Vision and Pattern Recognition
Summary
Finding a video based on a change you describe from another video is hard because videos are complex and big. The authors show that using large pre-trained AI models in smart steps—starting simple and getting more detailed only when needed—makes searching much faster and more precise. Their method doesn’t require extra training and works well on big video libraries. This helps computers find videos that match complex descriptions without slowing down too much.
What this means in practice
- •For video platform engineers: Implement scalable video search that understands detailed text-based edits without extra model training.
- •For media asset managers: Retrieve target videos matching nuanced text edits from large collections efficiently by combining fast embeddings with adaptive detailed verification.
Authors
Dmitry Demidov, Muhammad Zaigham Zaheer, Omkar Thawakar, Abdelrahman Mohamed Shaker, Rao Anwer
Abstract
Composed video retrieval (CoVR) searches a gallery for the target video that realizes a natural-language modification of a source clip. However, at gallery scale, this creates a fundamental tension: compact embeddings enable efficient, reusable search but can miss the transient actions, state changes, and subtle constraints that demand fine-grained video reasoning, whereas applying large multimodal models uniformly sacrifices scalability. To address these limitations, we propose that frozen foundation models should instead occupy complementary roles, with inference depth adapted to query difficulty. Based on this premise, we introduce \methodname{}, a framework for training-free \methodexpansion{}. Specifically, a composed-query embedding first searches reusable video-only gallery representations; uncertain queries undergo bounded reranking and candidate expansion; ambiguous edits trigger target-description generation; and only close leading candidates reach multimodal verification. To support these roles, frame selection, spatial resolution, and time cues are adapted to each stage. Across complete target-gallery evaluations, our method reaches state-of-the-art performance among training-free approaches, with 89.55 and 93.43 R@1 on Dense-WebVid-CoVR and CoVR-R, respectively (with more than +35\% and +25\% absolute margins to the closest counterpart). These results show that adaptively orchestrating foundation-model capabilities can combine scalable retrieval with fine-grained reasoning without task-specific training. The source code and all relevant guidelines are available on https://github.com/demidovd98/CoVRAGE.