Joint-conditioned method improves hand object interaction surface detection
Joint-Conditioned Stereo Surface Reasoning for Interaction Field Estimation
Computer Vision and Pattern RecognitionRobotics
Summary
Finding exactly where a hand touches an object in 3D images is hard, especially when parts are hidden. The authors treat this as guessing the exact spot on the object closest to each finger joint, while using clues shared among all fingers. They create a system called JSSR that looks at images from multiple views over time to better guess these contact points. This system also balances how much to trust the images or the geometry when things aren’t clear. Their method performed well in a competition measuring hand-object interaction estimation.
What this means in practice
- •For augmented reality developers: Generate accurate 3D hand-object contact points to improve realistic interaction in AR experiences.
- •For robotics manipulation teams: Estimate precise hand-object contact areas from stereo images to aid robot grasp planning and manipulation.
Authors
Yanlin Jin, Yifan Yang, Bowen Yang, Kai Zhu
Abstract
Predicting hand--object interaction fields requires locating the nearest object-surface point for each hand joint, often from small and partially occluded image regions. We view this task as joint-conditioned surface-endpoint estimation: each joint has its own nearest endpoint, while endpoints from the same hand can draw on shared local surface evidence. This structure motivates Joint-Conditioned Stereo Surface Reasoning (JSSR). A temporal-stereo network jointly predicts 3D joints, a direct interaction field, and per-view endpoint evidence. Calibrated candidate search evaluates endpoint hypotheses using joint-specific image compatibility and cross-view correspondence. A hand-shared candidate support lets joints draw on common surface evidence, and a learned residual gate controls the geometric correction when observations are ambiguous. Our system built on this method ranked third on the SHOW3D Interaction Field Challenge leaderboard.