Dual-side enhancements improve video segment retrieval without training
DSE-VTG: Dual-Side Enhancement for Training-Free Video Temporal Grounding
Computer Vision and Pattern Recognition
Summary
Finding the right part of a long video that matches a text description is hard and usually needs a lot of training. The paper introduces a new method that doesn’t require any extra training but still gets better at this task. They improve how videos are compared by looking at both single frames and groups of frames, and they also tweak the text description to better fit the video’s content. This approach works well even when the videos are different from the training examples used by other methods.
What this means in practice
- •For video content indexing teams: Improve locating relevant video segments from text queries without retraining models when video types change.
- •For media monitoring services: Enhance automated video clipping for text-based search in broadcast and online media with adaptable query understanding.
Authors
Zhuo Cao, Bingqing Zhang, Sen Wang, Xue Li
Abstract
Text-guided Video Temporal Grounding (VTG) aims to localize the relevant segments in an untrimmed video based on text queries, yet collecting dense temporal annotations and training task-specific models remain costly and brittle under distribution shift. Recent training-free VTG approaches mitigate this issue by directly matching pretrained vision-language representations, but they still face two fundamental information bottlenecks: frame-wise visual encoding overlooks temporal dynamics, while fixed query embeddings cannot resolve query ambiguity. To address these issues, we propose DSE-VTG, a \underline{D}ual-\underline{S}ide \underline{E}nhancement framework that addresses both without any task-specific training. On the visual side, Multi-scale Similarity Fusion (MSF) combines frame- and clip-level similarities into a unified, temporally aware similarity profile. On the textual side, Query-level Test-Time Adaptation (Q-TTA) optimizes a lightweight additive offset to adapt the query embedding to the video at test time, without finetuning the backbone or calling external large language models. Extensive experiments on three standard and two OOD benchmarks show that DSE-VTG achieves state-of-the-art performance among training-free methods. On Charades-STA, it improves mIoU over the strongest prior training-free method by 5.61 points. Under distribution shift, DSE-VTG reaches 50.86 mIoU on Charades-CG Novel-Word, surpassing the strongest supervised baseline by 2.76 mIoU. Our code will be released upon acceptance.