Dual enhancement improves video segment search without training
DSE-VTG: Dual-Side Enhancement for Training-Free Video Temporal Grounding
Computer Vision and Pattern Recognition
Summary
Finding the right parts of a video that match a text description is hard and usually needs lots of training with labeled examples. The researchers found a way to do this without training by improving how video and text are matched. They looked at both the video and the text more carefully: they combined information about individual frames and groups of frames in the video, and they adjusted the way the text is understood based on the video itself. This method works better than previous ones that don’t rely on training and even beats some methods that do use training, especially when the videos are different from the ones seen before.
Video temporal groundingVision-language modelsTraining-free methodsTemporal dynamicsQuery adaptationMulti-scale similarityZero-shot learningOut-of-distributionCharades-STAmIoU
Authors
Zhuo Cao, Bingqing Zhang, Sen Wang, Xue Li
Abstract
Text-guided Video Temporal Grounding (VTG) aims to localize the relevant segments in an untrimmed video based on text queries, yet collecting dense temporal annotations and training task-specific models remain costly and brittle under distribution shift. Recent training-free VTG approaches mitigate this issue by directly matching pretrained vision-language representations, but they still face two fundamental information bottlenecks: frame-wise visual encoding overlooks temporal dynamics, while fixed query embeddings cannot resolve query ambiguity. To address these issues, we propose DSE-VTG, a \underline{D}ual-\underline{S}ide \underline{E}nhancement framework that addresses both without any task-specific training. On the visual side, Multi-scale Similarity Fusion (MSF) combines frame- and clip-level similarities into a unified, temporally aware similarity profile. On the textual side, Query-level Test-Time Adaptation (Q-TTA) optimizes a lightweight additive offset to adapt the query embedding to the video at test time, without finetuning the backbone or calling external large language models. Extensive experiments on three standard and two OOD benchmarks show that DSE-VTG achieves state-of-the-art performance among training-free methods. On Charades-STA, it improves mIoU over the strongest prior training-free method by 5.61 points. Under distribution shift, DSE-VTG reaches 50.86 mIoU on Charades-CG Novel-Word, surpassing the strongest supervised baseline by 2.76 mIoU. Our code will be released upon acceptance.