Shot-aware matching improves text to video retrieval for multi-shot scenes
SALI: Shot-Aware Late Interaction for Cross-Shot Relation Matching in Text-to-Video Retrieval using Film-Grammar Knowledge
Computer Vision and Pattern RecognitionInformation RetrievalMultimedia
Summary
Videos often show interactions between people using multiple camera shots, like one showing Anna and another showing Mark. Usual methods that use just one combined video description can't capture these back-and-forth interactions well. The authors propose SALI, a method that looks at each shot separately and matches parts of the text query (like 'Anna' and 'Mark') to specific shots. This approach improves the ability to find videos based on descriptions involving interactions across different shots.
What this means in practice
- •For video search engine developers: Enhance search engines to better find video clips showing interactions filmed in multiple shots using subject-object aware matching.
- •For video editing software teams: Improve tools that tag and organize video scenes by recognizing interactions across alternating shots more accurately.
Authors
Toya Oyama, Rainer Lienhart, Shin'ichi Satoh
Abstract
Text-to-video retrieval usually represents a video clip by a single embedding. This embedding often loses important relations between people. E.g., an interaction "Anna confronts Mark" is regularly filmed as alternating shot and reverse shot of both (Fig. 1a). No single shot or averaged embedding over clip shots captures this relation. Thus, we propose SALI (Shot-Aware Late Interaction). It extracts the subject and object from a single-sentence query, and matches the query, its subject and object text embeddings against each visual shot embedding of a video clip. The matching operator is greedy max or optimal transport. A film-grammar penalty in fine-tuning adds a small, consistent shift. Built on CLIP4Clip-meanP, SALI keeps overall recall on par on Condensed Movies and ActivityNet while raising R@1 on multi-shot relation queries by 3 and 12 points, the most among all compared methods, and improves such queries on MSR-VTT at a cost of 1.4 R@1 overall.