Single head camera replaces wrist sensors for robotic hand control
A Head Camera Is (Almost) All You Need: An Electronics-Free UMI Glove Labelled and Observed by One Moving Camera
RoboticsArtificial Intelligence
Summary
Collecting data to teach robots how to use their hands traditionally requires heavy equipment on the user's wrist and complicated setups. The authors show that wearing one camera on the head can track a special glove with printed markers, replacing multiple devices and simplifying data collection. This method accurately tracks hand movements and helps train robot control policies almost as well as more complex systems. It improves the success rate of robotic tasks by combining demonstrations from the glove and traditional controls.
What this means in practice
- •For robotics engineers: Use a head-mounted camera and printed marker gloves to collect training data for robot hand control without complex sensors.
- •For quality assurance teams: Monitor robotic grasp quality and hand movements during manufacturing tasks using a single camera setup without extra wrist sensors.
Authors
Yosub Shin, Yuan Liu, Igor Molybog
Abstract
Handheld demonstration devices decouple data collection from the robot, but conventional designs burden the hand with a wrist camera, cables and a pose tracker (SLAM, an IMU, a VR controller, or a fixed external camera that the collector's body occludes), and burden the pipeline with as many data streams to timestamp, synchronise and cross-calibrate. We show that a single egocentric camera (USD 122) worn on the collector's head replaces the wrist camera, the pose tracker and their separate streams with one. It reads printed fiducials on an electronics-free glove (USD 22) against paper tiles taped around the workspace, giving the 6-DoF gripper pose and the jaw width from the same frames the policy will see; at deployment the same camera rides a low-cost (USD 120) arm beside the robot that mimics the collector's head from the robot's joint state. Against the robot's encoders the labels are accurate to 6.5 mm and 1.6 degrees median with 93-100% frame coverage, and a policy trained on them instead of encoders pays 3% in validation loss. Adding 194 glove demonstrations to 48 teleoperated episodes lifts rollout success from 10/25 to 17/25, level with 172 teleoperated episodes alone (18/25), and the full co-trained head-camera policy succeeds in 20/25 against 15/25 for the wrist-camera recipe, whose policies drop the object when the goal leaves their view, and 12/25 for a fixed camera.