ProxiDex improves multi-finger manipulation under uncertain contact
ProxiDex: Learning Dynamics-Guided Proximity Policy for Dexterous Manipulation
RoboticsArtificial Intelligence
Summary
Multi-fingered robot hands need stable ways to touch and hold objects, but it's hard to observe these contacts clearly because the hand blocks cameras and touch sensors are tricky to use. The researchers created ProxiDex, which uses 3D shapes of the area around the hand and object to guess how close they are and how this changes over time. This lets the robot better plan its moves even when vision isn't perfect. In tests with both simulations and real robots, ProxiDex helped robots handle new objects and unexpected situations more successfully than previous methods.
What this means in practice
- •For robotics engineers: Enable robots to maintain stable grasps and adapt actions despite limited or occluded visual feedback during object manipulation tasks.
- •For virtual reality developers: Use immersive proximity feedback for more realistic and responsive VR teleoperation involving robotic hands.
Authors
Yushan Bai, Boyu Zheng, Zhiyang Mao, Hongzheng Sun, Yuchuang Tong, En Li, Zhengtao Zhang
Abstract
Multi-finger dexterous manipulation relies on stable hand-object interactions, yet these interactions are partially observable in practice. Visual observations are often occluded by the hand, tactile sensors introduce hardware-specific modalities and calibration burdens, and existing policies rarely model how these cues evolve under actions, making them brittle under contact uncertainty. To address these, we present ProxiDex, a dynamics-guided proximity policy framework that treats hand-object proximity as an interaction state for dexterous manipulation. ProxiDex reconstructs interaction point clouds and converts geometric distances into proximity cues, forming a hardware-agnostic contact representation that provides immersive feedback during VR teleoperation. Built on this representation, ProxiDex learns action-conditioned proximity dynamics with a coupled forward-inverse design: future observation latents are predicted from actions, while proximity variations are decoded from latent changes. Leveraging these dynamics, ProxiDex adaptively reweights proximity tokens across manipulation phases and uses dynamics-consistency supervision to guide policy inference, stabilizing action generation under unreliable visual feedback. Simulation and real-world experiments demonstrate improved success rates and robustness over representative baselines across standard, unseen objects, and perturbation scenarios. Additional visualizations are available at https://proxidex.github.io/.