Point4D reconstructs long-term 3D motion tracks from videos
Point4D: Long-range 4D Motion Reconstruction
Computer Vision and Pattern Recognition
Summary
Tracking how points move in 3D over a long video is hard because existing methods only handle short clips well. The authors created Point4D, a model that predicts where each point goes through hundreds of video frames without needing to check the image repeatedly. By cleverly reusing information from frames where points are visible, Point4D builds detailed 3D paths more reliably. This lets it keep track of motion across long videos better than previous quick methods.
What this means in practice
- •For computer vision engineers: Improve long-term 3D tracking in video applications by integrating Point4D’s chunk-based trajectory prediction without repeated image matching.
- •For augmented reality developers: Use Point4D to maintain accurate 3D point motion over long video sequences for more stable AR overlay and environment understanding.
Authors
Minsik Jeon, Jay Karhade, Deva Ramanan, Shubham Tulsiani
Abstract
We introduce Point4D, a feed-forward model for 4D reconstruction of long-range video sequences. Point4D is able to reliably infer dense per-point 3D trajectories across multi-hundred-frame videos, unlike existing 4D methods that are limited to short input windows of at most a few dozen frames. A key innovation that enables this is our flexible 3D query-based motion decoder that decouples trajectory prediction from image-plane visibility. The predicted 3D endpoints are then directly re-queried in the next chunk without re-projection or matching. Furthermore, we show that extracting and reusing a visual descriptor from an arbitrary frame where the point is visible leads to better performance than relying solely on the source patch. Overall, Point4D achieves state-of-the-art performance across diverse long-video tracking benchmarks spanning over 200 frames and largely outperforms previous feed-forward 4D method. Project page: https://point-4d.github.io