Diffground improves zero shot video question grounding with semantic actions
DiffGround: Semantic Difference-Guided Zero-Shot Video Question Grounding
Computer Vision and Pattern Recognition
Summary
Video Question Grounding means finding the exact moments in a video that answer a question. This is hard because it requires understanding both the question and the video timeline, and existing methods often need lots of labeled examples. The authors created DiffGround, which doesn’t need those examples and uses large pre-trained language and vision models to focus on key video moments based on actions related to the question. They also developed a way to make the timing of these video segments more precise by comparing small differences in the video features. Their approach works better than past methods on popular tests.
What this means in practice
- •For video content analysis teams: Automatically locate relevant video segments answering natural language questions without needing manually annotated timelines.
- •For multimedia search engine developers: Improve search accuracy by using action-guided queries to retrieve precise video clips related to user questions.
Authors
Cheng-Han Lee, Amir Salimi, Hsiang-Wei Huang, Che-Chun Su, Min Sun, Cheng-Hao Kuo
Abstract
Video Question Grounding (VideoQG) requires both accurate question answering and precise temporal localization. Existing supervised methods rely on costly temporal annotations, while existing zero-shot approaches are primarily designed for Video Question Answering (VQA) and are not optimized for temporal grounding. We propose DiffGround, an action-driven zero-shot framework built upon frozen Multimodal Large Language Models (MLLMs). Our framework first predicts an answer-relevant action cue that transforms the input question into an explicit semantic target for grounding. Action Cue converts a reasoningoriented question into an action-centric retrieval query. Guided by the action cue, a Vision-Language Model (VLM) efficiently selects informative keyframes for reasoning. Finally, we introduce Semantic Diff Refinement, a training-free temporal refinement method that exploits semantic differences between neighboring SigLIP2 embeddings to recover complete action segments from sparse predictions. Extensive experiments on NExT-GQA and STAR show that DiffGround consistently outperforms existing zero-shot and supervised methods, establishing a new state of the art for Video Question Grounding.