System estimates hand and object contact points from stereo images
Joint-Conditioned Stereo Surface Reasoning for Interaction Field Estimation
Computer Vision and Pattern RecognitionRobotics
Summary
Estimating where a hand touches an object in 3D images is hard because parts of the hand and object can be hidden or unclear. The authors treat this as a task where each finger joint finds its closest point on the object’s surface, but joints also share clues from the surrounding area. Their method uses images from two cameras over time to guess the hand’s position, interaction areas, and possible contact points. They then check these guesses carefully using consistency between views and shared information across joints, improving accuracy when the images are ambiguous. Their approach performed well in a competition focused on this difficult problem.
hand-object interaction3D joint estimationstereo visionsurface endpoint estimationtemporal networkcross-view correspondenceinteraction fieldresidual gatingcalibrated candidate search
Authors
Yanlin Jin, Yifan Yang, Bowen Yang, Kai Zhu
Abstract
Predicting hand--object interaction fields requires locating the nearest object-surface point for each hand joint, often from small and partially occluded image regions. We view this task as joint-conditioned surface-endpoint estimation: each joint has its own nearest endpoint, while endpoints from the same hand can draw on shared local surface evidence. This structure motivates Joint-Conditioned Stereo Surface Reasoning (JSSR). A temporal-stereo network jointly predicts 3D joints, a direct interaction field, and per-view endpoint evidence. Calibrated candidate search evaluates endpoint hypotheses using joint-specific image compatibility and cross-view correspondence. A hand-shared candidate support lets joints draw on common surface evidence, and a learned residual gate controls the geometric correction when observations are ambiguous. Our system built on this method ranked third on the SHOW3D Interaction Field Challenge leaderboard.