Point4D enables tracking of 3D motion over hundreds of video frames
Point4D: Long-range 4D Motion Reconstruction
Computer Vision and Pattern Recognition
Summary
Tracking how objects move in 3D over a long video is difficult, especially when the video has hundreds of frames. The authors created Point4D, a system that can predict where points move in 3D space throughout very long videos without needing to look back at every frame. They do this by predicting the 3D end positions and then looking for those positions directly in the next part of the video, instead of relying on traditional image matching. This approach improves accuracy and allows for tracking motion over much longer video sequences than before.
4D reconstruction3D trajectoryvideo trackingmotion decoderfeed-forward modelimage-plane visibilityvisual descriptorlong-range trackingpoint cloudmulti-frame video
Authors
Minsik Jeon, Jay Karhade, Deva Ramanan, Shubham Tulsiani
Abstract
We introduce Point4D, a feed-forward model for 4D reconstruction of long-range video sequences. Point4D is able to reliably infer dense per-point 3D trajectories across multi-hundred-frame videos, unlike existing 4D methods that are limited to short input windows of at most a few dozen frames. A key innovation that enables this is our flexible 3D query-based motion decoder that decouples trajectory prediction from image-plane visibility. The predicted 3D endpoints are then directly re-queried in the next chunk without re-projection or matching. Furthermore, we show that extracting and reusing a visual descriptor from an arbitrary frame where the point is visible leads to better performance than relying solely on the source patch. Overall, Point4D achieves state-of-the-art performance across diverse long-video tracking benchmarks spanning over 200 frames and largely outperforms previous feed-forward 4D method. Project page: https://point-4d.github.io