Video point model forecasts 3D scenes and robot actions together
Learning the 4D World through Point Trajectory Forecasting
Computer Vision and Pattern Recognition
Summary
When multiple cameras watch the same scene, it’s hard to predict how things will move and look from any viewpoint. The authors created a system called the Video-Point Model (VPM) that uses videos from many cameras along with simple language instructions to predict future videos and track how points in the scene move in 3D over time. They trained and tested this system using simulated datasets with matched videos and point paths and showed it improves predictions of both appearance and motion. They also adapted the model to help control robots by including robot actions, which helped robots perform tasks better in simulations and real-life tests.
What this means in practice
- •For robotics engineers: Improve robot manipulation success by grounding control policies in 3D-aware video and point trajectory forecasts.$Commercial implications: Enables development of advanced robot controllers that predict and act in 3D space more accurately for industrial automation.
- •For computer vision developers: Generate consistent future videos from multiple camera views with tracked 3D points, improving multi-view forecasting accuracy.
Authors
Byungwoo Jeon, Sunghwan Hong, Dohyeon Kim, Jonghoon Lee, Jungwoo Park, Inhee Lee, Doosan Baek, Jinil Kim, Taehyun Lim, Hanbyul Joo, Marc Pollefeys, Saining Xie, Jinwoo Shin
Abstract
Reliable multi-view forecasting requires videos from different cameras to describe the same evolving 3D scene. To this end, we introduce Video-Point Model (VPM), a world model that conditions on multi-view video histories and a language instruction to jointly forecast future videos from arbitrary viewpoints and the 3D trajectories of persistent scene points. We extend a pretrained world model with camera-conditioned multi-view processing that integrates complementary observations across cameras, and add a trajectory head that predicts how physical points move through time. Each trajectory tracks a single point across all views, and supervising these trajectories alongside video generation therefore anchors appearance changes in every camera to a shared description of 3D motion. To support training and evaluation, we build VPM-Data, a simulation corpus pairing synchronized multi-view videos with persistent point trajectories, and VP-WorldBench, a forecasting benchmark spanning simulated and real environments. We further extend VPM to robot control by adding robot actions as a token modality in the generator. Action tokens share the point-supervised backbone and are denoised jointly with future videos, thereby grounding the policy in the same 3D-aware representation. On VP-WorldBench and existing 4D generation benchmarks, point supervision improves both video fidelity and 3D motion accuracy over a video-only counterpart, and in simulated and real-robot experiments, it also raises the manipulation success of the action-enabled model.