Hand and object motion generated smoothly from single start state
Learning Dynamic Grasping
RoboticsGraphics
Summary
Making robots or animations that can grasp objects realistically involves showing a hand moving, touching, and lifting things properly. This paper shows a way to create the entire motion of a hand grasping an object starting from just one initial position, without needing extra instructions. The authors separate the hand's movement from the object's motion and use a special prediction method to create smooth motion sequences. Their approach works better and faster than previous methods, especially for new objects, but achieving perfect contact with unfamiliar objects still needs more work.
What this means in practice
- •For character animation teams: Generate realistic dynamic hand-object grasping motions from a single initial frame to enhance animation workflows.
- •For extended reality developers: Create more believable hand-object interactions for VR and AR applications without requiring complex input conditions.
Authors
Nga Kiu Kelly Ho, Yilin Liu, Hao Xu, Niloy J. Mitra
Abstract
Synthesizing believable 4D hand-object grasping sequences is important for character animation, extended reality, and embodied agents. Grasping requires a hand to approach, contact, and lift or manipulate an object coherently. Existing synthesis methods often model only the final static grasp or rely on additional conditions, such as language, target grasps, or object trajectories, to specify the interaction. We instead study the underconstrained problem of generating a complete grasping motion from a single initial hand-object state. Our framework decouples hand-motion generation from object-motion recovery. Given an object point cloud in the initial wrist frame and the initial parameterized hand configuration, conditional flow matching predicts a sequence-level contact map and a hand pose at each normalized sequence time. This time-indexed formulation avoids the error accumulation of autoregressive rollout while representing the complete motion. We then infer an attachment frame and recover rigid-object motion through training-free, contact-guided weighted registration. We evaluate on held-out DexYCB objects and unseen object geometries reconstructed from in-the-wild videos by Open-CHOIR. Compared with LatentHOI, our method produces smoother hand and object motion in every setting, matches or improves on its contact and penetration metrics in the wild under a shared initialization, and is about 30x faster per generated frame. In a 15-participant study, our motions were preferred in roughly two thirds of (forced-choice pairwise) comparisons. Precise contact closure on novel geometry, however, remains a key open challenge.