Physics aware method improves hand object interaction reconstruction accuracy
DynamicHOI: Coupled Dynamics for Physics-aware HOI Reconstruction
Computer Vision and Pattern Recognition
Summary
Hand-object interactions in videos can look believable but may not follow the rules of physics, leading to unrealistic movements. The authors present DynamicHOI, a new approach that combines visual information with physics principles to make these interactions more realistic. Their method uses knowledge about forces and motion to refine how hands and objects move together. This results in more accurate reconstructions that better match real-world physics.
What this means in practice
- •For robotic manipulation teams: Improve robot grasping and object handling by training with more accurate physics-based hand-object interaction models extracted from videos.
- •For augmented reality developers: Create more physically consistent virtual hand and object interactions in AR applications using physics-aware reconstruction.
Authors
Wenliang Guo, Zhanbo Huang, Yu Kong
Abstract
We study hand-object interaction (HOI) reconstruction from monocular RGB videos, where partial observations can produce visually plausible yet mechanically inconsistent trajectories. Existing methods mainly enforce visual and geometric agreement, leaving the underlying interaction dynamics insufficiently constrained. We propose DynamicHOI, a physics-aware HOI reconstruction framework combining geometry-grounded diffusion refinement with coupled hand-object dynamics. Geometry spatially grounds visual evidence for trajectory refinement, while articulated inverse dynamics and Newton-Euler dynamics derive hand generalized forces and object wrenches for dynamics-level supervision. We further couple hand and object dynamics through contact-force transfer and recover active hand actuation as an interaction-level physical quantity. We formulate its empirical magnitude distribution into a probabilistic prior that penalizes unlikely actuation and suppresses mechanically implausible reconstructed motion. Experiments on three HOI datasets show consistent improvements in both hand and object reconstruction. The reconstructed trajectories further benefit downstream applications including hand world-model generation and robotic manipulation learning, demonstrating the value of physics-aware HOI modeling beyond reconstruction.