Interaction-aligned method improves video segment detection for action-entity queries
IAE-VTG: Interaction-Aligned Action-Entity Video Temporal Grounding
Computer Vision and Pattern RecognitionMultimedia
Summary
Finding the exact part of a video that shows a specific action happening with a certain object or person is hard because many methods only look for the action and object separately. The authors created a system called IAE-VTG that carefully checks whether the action and the entity actually happen together in the video. Their method breaks down the query into parts about the action and the entity, matches these with video features, and uses this combined understanding to better identify the correct video segment. Tests show it works better especially when the video shows similar actions or objects multiple times.
What this means in practice
- •For video content editors: Pinpoint exact video clips showing specific actions by certain subjects to speed up editing and highlight creation.
- •For surveillance system developers: Automatically locate moments where a particular person performs a given action, improving event monitoring accuracy.
- •For online video platforms: Enhance video search by precisely matching user queries involving actions and entities to correct segments within videos.$Commercial implications: Enables advanced search features that improve user engagement and content discoverability, creating commercial value for platform providers.
Authors
Shiwen Zhao, Qi Zhang, Sezer Karaoglu, Theo Gevers, Martin R. Oswald
Abstract
Video Temporal Grounding (VTG) localizes the video segment that matches a natural-language query. Many queries describe an action performed by a particular entity. Existing methods often encode the query as a whole or use general video-text interactions, without explicitly checking whether the action and entity occur together. They may therefore select a segment that contains both concepts but not the event described by the query. We propose Interaction Aligned Action-Entity Video Temporal Grounding (IAE-VTG), which models this rela?tionship at both the representation and training assignment levels. First, the Fine-grained Disentangled Interaction Module (FDIM) separates action and entity related query information and aligns it with complementary motion and appearance features. It then combines token-level interactions to build representations that capture the relationship between the action and entity. Second, Interaction-Sensitive Assignment (ISA) adds this interaction evidence to bipartite matching, so training targets are selected using both temporal overlap and semantic compatibility. This reduces supervision from temporally plausible but semantically incorrect proposals. Experiments on QVHighlights, Charades?STA, and TACoS show that IAE-VTG consistently improves strong baselines and achieves competitive or state-of-the-art performance on standard grounding metrics. Additional analyses show that the method is especially effective when similar actions or entities appear at multiple times and produces more reliable assignments for complex events.