PointZero predicts 3D object movements without robot action labels
PointZero: 3D Point Track Completion for Learning Transferable 3D Dynamics
Computer Vision and Pattern RecognitionRobotics
Summary
Teaching machines to understand how objects move in 3D usually needs detailed robot actions during training, which limits the types of videos available. The authors created a method called PointZero that can predict how every point on an object will move in the future using only one depth image and some partial movement information, without needing robot data. They trained it on a huge set of computer-generated scenes with different types of objects, and it learned a general sense of 3D motion. This makes it helpful for tasks like predicting how objects will move when acted on or copying robot actions.
What this means in practice
- •For robotics engineers: Improve robot manipulation by fine-tuning PointZero to predict 3D object motions conditioned on robot movements.
- •For autonomous system developers: Use PointZero’s pre-trained model to better predict object movements in scenes without requiring robot action annotations.
Authors
Bardienus P. Duisterhof, Kaifeng Zhang, Adam Hung, Bowen Wen, Stan Birchfield, Yunzhu Li, Deva Ramanan, Jeffrey Ichnowski
Abstract
World models endow perceptual systems with the ability to predict how scenes evolve under interaction. They are most beneficial when trained on diverse volumes of data, to instill a rich prior into downstream applications. Existing methods typically require robot action labels to learn action-conditioned 3D dynamics, which excludes web video data from the training pool. We study 3D point track completion as a pre-training objective for learning transferable 3D dynamics without robot data. Given a single RGB-D observation and sparse partial 3D trajectories (tracks), we predict future 3D tracks of all observed points. We show this objective produces a rich 3D dynamics prior, without requiring robot action labels. We contribute a diverse dataset of 2.9 million synthetic frames spanning deformable, articulated, and rigid objects, and use it to train PointZero. We show that a flexible and expressive transformer, PointZero, outperforms prior methods on the same data. We demonstrate the utility of our pre-training objective by post-training PointZero for two downstream applications: (1) action-conditioned 3D dynamics prediction and (2) imitation learning. When fine-tuned to condition on end-effector pose, PointZero outperforms the baselines on the recent PGND 3D dynamics benchmark. When fine-tuned to predict robot actions and 3D tracks, PointZero outperforms or matches the baselines on 6/7 simulated and real-world robot manipulation tasks. We furthermore evaluate training PointZero from scratch to isolate the benefits of our proposed architecture from those of our proposed pre-training objective and dataset. We release the dataset, checkpoints, and full training recipe.