Text meets video with better focus and detailed query matching
Text-Video Retrieval via Multi-Dimensional Saliency Assessment and Granularity-Aware Query Decomposition
Information Retrieval
Summary
Finding the right video based on a text description is tricky because videos have many moments and details, and text summaries often miss this complexity. The authors created a method that picks out important parts of a video by looking at different visual cues and scores. Their approach also breaks down the text into smaller parts to match video details at multiple levels, like whole scenes or tiny patches. This leads to better matching between the text and the correct parts of videos.
What this means in practice
- •For video search developers: Enhance video search engines by identifying and using key video frames and patches alongside multi-layered text queries for more precise retrieval results.
- •For media content curators: Automatically align and tag video content with detailed textual descriptions at various levels, improving content organization and recommendation.
Authors
Shuquan Wei, Xi Chen, Xu Chen, Xiangyang Jia
Abstract
Text-video retrieval, which aims to bridge visual and textual modalities by learning a joint embedding space, has become a crucial task in multimodal intelligence. Despite extensive efforts to mitigate visual redundancy, previous methods typically rely on a single-aspect criterion to assess visual importance, overlooking the multifaceted spatiotemporal nature of video. In addition, encoding text into a single global embedding to align with videos compresses temporal events and spatial entities into a unified representation space, further aggravating cross-modal misalignment. To address these issues, we propose MMTI, a method that jointly mitigates visual redundancy and enables multi-grained text-video interaction to achieve accurate multi-grained semantic alignment. Specifically, a key feature selection (KFS) mechanism adaptively identifies and aggregates informative frames and patches by jointly evaluating multi-dimensional saliency and learnable importance scores, effectively compacting dense visual features and mitigating visual redundancy. Furthermore, our proposed multi-grained text-video interaction module (TVIM) employs a dynamic gating mechanism to decompose the text query into sentence, frame, and patch queries (SFP), enabling multi-grained text-video alignment. Complementary alignment at different granularities is thereby achieved. Extensive experiments on four standard benchmarks demonstrate that our method outperforms state-of-the-art methods.