3D scene tracker monitors all points over long videos efficiently
TrackEverything: Long Horizon Dense Tracking via De-Duplicating 3D Scene Representations
Computer Vision and Pattern RecognitionArtificial IntelligenceRobotics
Summary
Tracking points in videos usually involves a tradeoff between following a few points for a long time or many points over a short clip. The authors designed a system called TrackEverything that can track nearly every point visible in a video over very long periods by using 3D models of the scene. They avoid repeating data by combining duplicate points and focus computational effort only on points that move. This approach lets them track thousands of points for hundreds of seconds using reasonable computer memory.
What this means in practice
- •For autonomous vehicle developers: Improve object and scene tracking accuracy over long video sequences to enhance perception and navigation safety.
- •For robotics engineers: Enable robots to monitor every visible surface point in a scene continuously for complex tasks requiring long-term spatial awareness.
Authors
Ayush Jain, Sreeharsha Paruchuri, Ishita Gupta, Fan Zhang, Tanner Schmidt, Jakob Engel, Katerina Fragkiadaki, Adam W. Harley
Abstract
Existing point tracking models face a fundamental tradeoff: they can either track a sparse set of query points over long horizons, or track all points across only short clips. We introduce TrackEverything, a 3D point tracker that breaks this trade-off by representing videos as persistent 3D scene tracks in world coordinates. Grounded in the insight that videos are 2D projections of an underlying 3D world, TrackEverything decouples model complexity from video duration, allowing it to scale with unique physical scene geometry instead. Our approach introduces three key innovations. First, we employ a voxelization-based de-duplication mechanism at sliding-window boundaries to merge co-located tracks, preventing repeated observations of the same surface from redundantly accumulating. Second, we decompose tracking into an endpoint refiner that predicts each point's destination and static-versus-dynamic classification, followed by a lightweight trajectory refiner that decodes dense trajectories exclusively for dynamic points. Third, we propose 3D WAFT, replacing memory-prohibitive 4D correlation volumes with efficient feature sampling in the scene cloud. To the best of our knowledge, TrackEverything is the first 3D tracker capable of tracking all visible points across videos exceeding 1000 frames within 40 GB of GPU memory. On TAPVid-3D, TrackEverything outperforms all open-source all-frame dense 3D trackers by more than 20% APD on short clips, while remaining competitive with state-of-the-art sparse trackers on long sequences, despite tracking far more points.