Human limb coupling enables video to exoskeleton motion mapping
ExoBridge: Learning a Bare Hand to Hand-Worn Exoskeleton Mapping through Human Limb Coupling
Robotics
Summary
It’s hard to capture detailed hand motion and touch data while watching someone use their bare hands in videos. The authors created ExoBridge, which uses the linked motion of both hands together: one hand wears a sensor-packed robotic glove while the other hand stays bare and is recorded on video. By learning from this paired data, the system can predict the robotic glove’s movements and tactile signals just by watching video of a bare hand. This helps connect human hand videos to robot hand control, potentially improving robot skill learning from uninstrumented video.
What this means in practice
- •For robotics engineers: Train dexterous robot hands from ordinary videos by converting bare hand motion into exoskeleton states for robot control.
- •For prosthetics developers: Use coordinated hand motion to infer tactile and motion signals for sensorized prosthetic devices without requiring direct instrumentation on the intact hand.
Authors
Ruitong Tian, Xianyao Li, Noah B. Wilson, Fang Xu, Eric Jing Du
Abstract
Human video offers a scalable source of experience for dexterous robot learning, but obtaining motion and tactile supervision while preserving bare hand interaction remains challenging. We present ExoBridge, a framework that leverages human limb coupling to learn a bridging function from bare hand video to the motion and tactile state of a sensorized exoskeleton. Our central idea is to use coordinated bimanual behavior to connect an uninstrumented visual source with a measured manipulation interface. During collection, one hand remains bare and provides visual observations, while the opposite hand wears the exoskeleton and supplies synchronized motion and tactile measurements. These paired demonstrations train a temporal visual model to predict fingertip contact, continuous tactile intensity, and relative encoder motion from bare hand video alone. The exoskeleton defines an intermediate state space whose motion coordinates are linked to a dexterous robot hand through existing calibration. Evaluation on 1,215 demonstrations across four manipulation tasks uses held out collection sessions and yields a pooled any contact AUROC of 0.916 and a Pearson correlation of 0.790 for tactile intensity. The learned bridge also predicts relative changes in exoskeleton configuration from bare hand video. These results demonstrate that human limb coupling can turn exoskeleton measurements into supervision for bare hand video, establishing a learned bridge between human visual demonstrations and a robot oriented manipulation interface.