Unified model predicts 3D hand and object motion from videos
Grasping the Future: Joint 3D Hand-Object Forecasting From Internet Videos
Computer Vision and Pattern Recognition
Summary
People often move their hands and objects together when using things. Predicting how hands and objects will move next can help machines understand and imitate human actions. The authors created a model that learns this from videos found on the internet by figuring out how hands and objects moved in past actions and guessing what will happen next. Their model predicts detailed 3D movements of hands and objects, including finger positions and wrist motion, and even works well with incomplete information. They tested it against existing tools and found it performs better at predicting future hand-object interactions.
What this means in practice
- •For robotics engineers: Program robots to manipulate objects by predicting human-like hand and object movements from video data.$Commercial implications: Enables development of robot grasping systems that imitate natural human interactions, useful for manufacturing and service robots.
- •For augmented reality developers: Create AR experiences that anticipate user hand gestures and object manipulation for more responsive interfaces.
Authors
Rustin Soraki, Matthew Wallingford, Ryan Aby, Brandon Chee, Jason Ren, Ali Farhadi, Homanga Bharadhwaj, Roozbeh Mottaghi
Abstract
To interact with an object, we need to anticipate both how it will move and how our hands should move with it. Human videos capture these relationships across everyday activities, offering a natural source for learning to predict future interactions. Learning from them requires recovering motion from completed interactions and using it to anticipate what could happen next. We present $\textit{Grasping-the-Future}$, a unified generative model of joint 3D hand-object motion, including wrist motion, finger articulation, and object position and orientation. Through masked spatiotemporal completion and flow matching, we train a single model for several prediction tasks: forecasting future interactions, predicting hand motion given object motion or vice versa, generating both trajectories toward a desired final object pose, and completing partially observed interactions. We explicitly model hand-object relations and incorporate contact constraints to help preserve grasp relationships during motion. To obtain 3D supervision from internet videos, we develop a scalable pipeline that combines pretrained perception models with joint geometric optimization to reconstruct completed interactions. We combine these recovered trajectories with annotated interaction datasets as training data for all tasks. We demonstrate strong performance across all tasks, outperforming released forecasters on datasets with 3D annotations. Moreover, we show geometric and physical plausibility of the predicted trajectories by retargeting them to a robotic embodiment. Videos are available at https://grasping-the-future.github.io.