PhotoHOI: Synthesizing 3D Hand-Object Interactions from a Single RGB Photograph
2026-08-03 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors focus on making 3D animations of hands interacting with objects from just one photo and a simple text instruction. They created a system called PhotoHOI that understands the picture and instruction, figures out the object and how it should move, then generates realistic hand motions that work even with new objects and scenes. Their method improves how hands touch objects and avoids unnatural collisions compared to older methods. Tests with different datasets and real photos show their system handles various instructions and objects well.
Hand-object interaction3D synthesisVision-language modelCollision avoidanceHand pose estimationContact modelingAffordance learningScene reconstructionOpen-vocabulary instructionsGRAB dataset
Authors
Zhenhao Zhang, Jiajun Zhang, Wei Min, Yebin Liu
Abstract
Hand-object interaction (HOI) is a fundamental human behavior with broad applications in AR/VR, digital humans, and embodied interaction. Existing methods typically require predefined object geometry, object trajectories, or task-specific conditions, limiting their use with natural real-world inputs. To address this, we study a more practical problem of synthesizing 3D hand-object interaction sequences from a single RGB photograph and an open-vocabulary language instruction, and introduce PhotoHOI. PhotoHOI first uses a vision-language model to parse the input image and instruction into a structured task specification, including the interaction object, target region, and spatial relation. It then recovers a compact task-relevant 3D scene and plans a smooth collision-aware object trajectory based on the recovered object states, support relations, and surrounding scene geometry. To synthesize hand motion that generalizes to real-world photographs and unseen objects, it learns transferable task-conditioned contact and contact-conditioned grasp priors from large-scale affordance and HOI data. The grasp is further refined in a learned latent space, constraining the optimization to a plausible hand-pose manifold. Experiments on GRAB and H2O demonstrate improved contact quality and reduced penetration over representative baselines. Results on real-world photographs further demonstrate higher task success and scene consistency, together with generalization to unseen objects and open-vocabulary instructions.