Video reasoning method tracks objects and events across frames
Think Visually Across Space and Time: Grounded Reasoning for Video Segmentation
Computer Vision and Pattern Recognition
Summary
Tracking specific objects and events over time in videos is hard because understanding both where things are and what happens requires linking spatial and temporal clues. The authors propose a method called Temporal Chain of Video Primitives (TCoVP) that breaks down video reasoning into a sequence of visual clues connected with text, making it easier to find the right object and predict its path. This approach helps models better link the language describing the target with the actual visual evidence across video frames. Their experiments show this method improves performance on several video segmentation tasks.
What this means in practice
- •For video analytics teams: Improve tracking of complex events and objects in surveillance footage by grounding reasoning in both space and time across frames.
- •For autonomous vehicle developers: Enable systems to better identify and follow relevant objects and their actions over time for safer navigation.
Authors
Junwei Wen, Deshui Miao, Guangming Lu, Ming-Hsuan Yang, Xin Li, Wenjie Pei
Abstract
Video Reasoning Segmentation (VRS) requires a model to identify an implicitly described target and predict its pixel-level mask trajectory, while resolving the spatial relations and temporal events that determine the answer. Text-only Chain-of-Thought (CoT) can explain a target choice without making the underlying visual references explicit, leaving a \emph{video reference gap} between linguistic inference and the evidence distributed across frames. We introduce \textbf{Temporal Chain of Video Primitives (TCoVP)}, a spatiotemporally grounded form of Video CoT that organizes query-relevant observations into a chronological reasoning chain. Each primitive associates an entity reference with temporal support and timestamped spatial evidence, and the chain interleaves these references with language to guide reasoning about which entity is relevant, when the queried event holds, and where its evidence appears. The selected target primitives then provide box prompts and temporal support for video mask prediction. We construct TCoVP-SFT-15k and TCoVP-RL-13k to learn this representation through supervised fine-tuning and chain-aligned reinforcement learning. Extensive experiments across five VRS benchmarks and three model backbones demonstrate the strong performance of TCoVP, highlighting the effectiveness of organizing spatial and temporal evidence within the reasoning process. All models and code will be publicly released. Project page: \url{https://victor-wjw.github.io/TCoVP/}