Video object paths matched with text using detailed embeddings
VTrajEmbed: Aligning Text with Video Object Trajectories via Spatiotemporal Embeddings
Computer Vision and Pattern Recognition
Summary
Matching descriptions to specific moving objects in videos is tough because videos usually mix all objects into one summary. The authors created VTrajEmbed, which breaks videos down into individual object paths and links each path to matching text. They use two ways of encoding video frames to capture both movement and scene details, plus a special training method to avoid mistakes in matching. Their model performed better than previous ones at finding objects from text and vice versa. They also built a new test set to check how well models do at linking text to object movements.
What this means in practice
- •For video content indexing teams: Improve search for specific objects or actions in videos by linking text queries directly to object movements across frames.
- •For surveillance system developers: Enhance detection and retrieval of specific objects or events from surveillance footage by matching textual descriptions to object trajectories.
Authors
Ruizhi Chen, Zining Fan, Yudi Guo, Jiang Liu, Zhiyu Lin, Zhongjiang He, Shouqian Shi, Hao Sun
Abstract
Fine-grained alignment between text and video object trajectories is a core challenge in multimodal understanding. Existing embedding models excel at global retrieval, yet their single global representation entangles multiple objects and their spatiotemporal dynamics, limiting direct object-level grounding. To address this issue, we propose VTrajEmbed, an embedding model that decomposes a video into object trajectories and aligns each trajectory embedding with text for object-level grounding through retrieval. Our main contributions: (1)Slow-Fast Trajectory Encoding, a dual-branch architecture that establishes trajectory representations by densely sampling low-resolution frames for high-frequency motion cues and sparsely selecting high-resolution frames for scene context, aggregated by a shared LLM backbone; (2)Mask-Guided Contrastive Learning, a training objective built upon these representations that couples an intra-group loss and a batch-level bidirectional loss under a relation mask, excluding unverified relations and preventing confirmed matches from becoming false negatives; (3)Text2Traj-Bench, a trajectory-level text-video benchmark with model-assisted construction and human-verified query--trajectory relations. Experimental results show that VTrajEmbed-5B improves text-to-trajectory and trajectory-to-text R@1 over Qwen3-VL-Embedding-8B by 15.48 and 18.05 percentage points, respectively, and achieves 49.48% average R@1 across five global retrieval benchmarks, compared with 42.86% for Qwen3-VL-Embedding-8B.