Query-guided models improve object and action spotting in videos
Where, How, and When: Query-Guided Grounding for Video Understanding
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Understanding videos involves finding where objects are, how they interact, and when actions happen. The authors created new methods that use guided queries to locate unseen objects and few examples of objects over time. They also developed ways to find relationships between things in videos and to retrieve specific moments by matching video sequences. These methods work together to help machines better understand what is happening in videos at different levels.
What this means in practice
- •For video analytics engineers: Identify and track unseen or rare objects and their interactions in video streams using query-based grounding models for improved event detection.
- •For video editing and search teams: Retrieve precise moments or segments in large video collections by matching video-to-video or text-to-video queries with temporally aligned features.
Authors
Yogesh Kumar
Abstract
Understanding video content requires reasoning at multiple levels of granularity, from identifying objects to interpreting their interactions and localizing when specific actions occur. This thesis studies three query-guided grounding tasks that answer where key objects are (object-level grounding), how they interact (relationship-level grounding), and when specific moments occur (moment-level grounding). For object-level grounding, we present QDETRv, a query-guided extension of the Detection Transformer for one-shot localization of unseen objects in videos, which combines query-context fusion, a memory module for temporal consistency, and unsupervised pretraining on synthetic object trajectories. We further propose the Temporal Object-Aware Vision Transformer (TOViT) for few-shot video object detection, which builds on a language-aligned open-set vision encoder and propagates high-confidence object features across frames. For relationship-level grounding, we introduce few-shot referring relationships in videos: given a (subject, predicate, object) query and a few support videos sharing the predicate, the entities are localized by optimizing a T-partite random field using a meta-learned relation network with query-conditioned translational embeddings, Global Semantic Aggregation, Local Localization Aggregation, and belief-propagation-based message passing. For moment-level grounding, we propose the Moment Alignment Transformer (MATR) for video-to-video moment retrieval, which uses dual-stage Dynamic Time Warping for pre- and post-fusion alignment, learnable queries for boundary prediction, and self-supervised pretraining with temporal augmentations; MATR also extends to text-to-video moment retrieval with performance comparable to, or better than, the state of the art. These contributions advance query-guided video understanding across spatial, relational, and temporal grounding.