Method tracks and reconstructs 3d shapes from single videos
AgentSTAR: Agentic Shape Tracking and Reconstruction from Monocular Videos
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Tracking and understanding moving objects in videos can be hard, especially when they change shape or get hidden behind other things. The authors created a method that builds a 3D model of the object and its parts and then improves its guess about where the object is over time by comparing rendered images to the video. This approach avoids relying on matching pixels directly, which helps it handle big movements and occlusions better. Their tests show it works better than previous methods at tracking both objects that move rigidly and those that bend or articulate.
What this means in practice
- •For augmented reality developers: Improve tracking of complex moving objects in AR by using 3D models with kinematic structure instead of pixel matching.
- •For robotics engineers: Enable robots to better track and understand articulated objects from single camera video for manipulation and interaction tasks.
- •For visual effects studios: Use the method to reconstruct and track 3D shapes in videos for more accurate CGI integration of moving and articulated objects.$Commercial implications: Makes possible commercial VFX tools that reconstruct object shapes and motion accurately from standard video footage.
Authors
Kirill Mazur, Nikita Karaev, Matthew Chang, Jitendra Malik, Nur Muhammad "Mahi'' Shafiullah
Abstract
In this work, we present a method for shape reconstruction and tracking from video via agentic analysis-by-synthesis. Unlike prior methods which first estimate dense pixel correspondences and then recover object motion from them, our method infers a structured 3D object model, including its geometry and kinematic structure, and uses this model to optimise object track estimates over time. In our optimisation loop, a Vision-Language Model (VLM) agent iteratively refines shape or generalised pose through a render-and-compare loop, combining coarse visual reasoning with numerical pose optimisation for precise state estimation. This structured formulation enables our method to track through large motion, articulation, and severe occlusion without relying on pixel-matching objectives. Quantitatively, on ARCTIC, our method substantially outperforms state-of-the-art 3D point-tracking baselines for articulated objects, and on HOT3D it outperforms all evaluated rigid-object tracking baselines.