GOLF achieves first place in hand object interaction estimation challenge
GOLF: Global Observation with Local Focus for Calibration-Aware Stereo Interaction Field Estimation
Computer Vision and Pattern Recognition
Summary
Estimating how hands interact with objects in 3D space is tricky, especially when using stereo camera views from a wearable device. The authors present GOLF, a new method that combines global and local information along with special geometry to predict the closest points on objects to various hand joints. This method outperforms others in a challenge, achieving the best accuracy in locating these interaction points. The approach uses advanced vision models and geometric reasoning to improve predictions.
What this means in practice
- •For augmented reality developers: Improve hand-object interaction tracking in AR devices using synchronized stereo camera input and precise 3D interaction field estimation.$Commercial implications: Enables more accurate and responsive hand tracking features in consumer AR headsets for interactive applications.
- •For robotics engineers: Enhance robot perception systems to better understand human hand-object manipulation in shared workspaces using egocentric stereo input.
Authors
Minqiang Zou, Riqiang Jin, Zhi Lv, Dong Luo, Lianghai Tian, Zhenyu Zhao, Qi Xu, Tong Wu, Mochen Yu, Yao Tang
Abstract
We present GOLF, the first-place solution to the SHOW3D Interaction Field Estimation Challenge at HANDS@ECCV 2026. Given synchronized egocentric stereo views, the task is to predict a 3D vector from each of 21 hand joints to the closest point on the manipulated object. GOLF combines dense global context, locally sampled hand/object evidence, and common-frame Plücker-ray geometry. We adapt DINOv3 ViT-H+/16 with LoRA and trainable LayerNorm parameters, then jointly decode both interaction fields. Our primary model achieves an official score of 27.61 and a mean ADE of 27.96 mm on the hidden test set. An equal-weight ensemble with a complementary directly fine-tuned variant improves these results to an official score of 27.47 and a mean ADE of 27.82 mm, securing first place.