Vision language models learn to track objects in 3d over time
Trace4D: Grounded Spatiotemporal Reasoning via 4D State Sequences
Computer Vision and Pattern Recognition
Summary
Understanding how objects move and change position in 3D space over time is hard for AI models that combine images and language. The authors introduce Trace4D, a way to train these models to predict sequences of object states in 3D plus time (4D) and to use this information to reason and answer questions. They collected a large dataset from real videos to train and test their method. The result is improved accuracy in understanding how objects relate and change through time in videos without needing extra external tools.
What this means in practice
- •For video analytics teams: Improve automated systems to track and reason about objects moving in 3D over time from video data without extra sensors.
- •For robotics developers: Enhance robot perception modules to understand evolving object arrangements and relations in real-world settings from camera feeds.
Authors
Wooseok Jang, Seungryong Kim, Yan Lu, Lei Ji
Abstract
Understanding how objects and their spatial relationships evolve in 3D space over time remains challenging for vision-language models (VLMs), particularly under changing viewpoints. Recent approaches incorporate geometric information into reasoning, but explicit supervision for temporally ordered 4D state evolution and its use as intermediate evidence remains limited. We introduce \textbf{Trace4D}, which trains VLMs to predict 4D state sequences and reason over their inter-timestep changes alongside temporal visual observations. Its 4D-grounded chain-of-thought (CoT) explicitly represents 4D states, analyzes temporal visual context, and connects both as evidence for the answer. We construct Trace4D-Train from in-the-wild videos with 600K 4D perception examples and approximately 70K grounded QA examples, and train with joint supervised fine-tuning followed by geometry-aware GRPO. At inference, Trace4D predicts 4D states and reasoning directly from video without reference annotations or an external geometric model. Experiments on Trace4D-Bench and VLM4D show improved QA accuracy over the base VLMs and benefits from reasoning over 4D state changes and temporal visual observations.