KineFuse: Kinematic-Aware Haptic Fusion for In-Hand Occluded-Object Pose Tracking
Robotics
Summary
The authors study how to better track an object’s position and orientation when fingers holding it block the camera view. They combine touch and force sensor data with a visual tracker using a special method that understands finger movements. Their tests show that this method works best in longer tracking sequences and helps improve robotic hand tasks like turning objects. They also share real-world examples to demonstrate their approach.
Authors
Chanyoung Ahn, Jaesung Lee, Sungwoo Park, Donghyun Hwang
Abstract
Dexterous in-hand manipulation requires continuous 6D pose tracking, yet the manipulating fingers inevitably occlude the object from the camera. We study how to structure the sparse haptic signals already available on multi-fingered hands, including proprioception, proximal force/torque, and binary contact, to complement a pretrained visual pose tracker under occlusion. We propose a kinematic-aware finger-level encoder and systematically compare it against four alternative designs through three levels of evaluation: per-frame refinement, sequential open-loop tracking, and closed-loop manipulation. Our experiments reveal that (i) per-frame evaluation cannot distinguish encoder quality, while sequential tracking amplifies architectural differences by up to 15 times; (ii) the structured encoder learns task-specific cross-modal gating, using vision exclusively for translation and dedicating one attention head to haptics for rotation, without explicit supervision; and (iii) compact finger-level tokenization with 4 tokens outperforms both flat fusion and joint-level representations, which suppress vision through norm dominance. We validate that improved tracking yields higher success in a downstream reorientation task and provide qualitative real-world demonstrations. Our project page is available at https://cold-young.github.io/kine-fuse/.