Pixel-UMI learns whole-body robot tasks from human demonstrations
Pixel-UMI: Learning Environment-Aware Whole-Body Manipulation from Sensorized Human Demonstrations
RoboticsArtificial IntelligenceComputer Vision and Pattern RecognitionMachine Learning
Summary
Robots can learn how to use their whole bodies to perform tasks by watching humans do them, but humans and robots move differently, which makes this hard. The authors created Pixel-UMI, a system that helps robots figure out how to match human movements while avoiding collisions and dealing with tricky viewpoints. To improve learning, the system also asks humans to correct it when it makes mistakes during practice. This helps robots handle complicated tasks like moving objects carefully and working with two hands.
What this means in practice
- •For robotics engineers: Enable robots to learn complex whole-body manipulation tasks directly from human demonstrations while adapting to environmental constraints.
- •For industrial automation teams: Improve robots’ ability to handle deformable or articulated objects in constrained spaces with real-time human corrections.
Authors
Jason Jingzhou Liu, Tony Tao, Deepak Pathak, Ruslan Salakhutdinov
Abstract
Sensorized human demonstrations offer scalable data for mobile manipulation, but embodiment differences complicate policy learning. We present Pixel-UMI, a pipeline for learning environment-aware whole-body policies from human data. We first introduce Environment-Aware Retargeting (EAR), which combines reconstructed scene geometry with demonstrated head and gripper trajectories to determine robot posture and generate collision-aware joint and base targets. Delay-calibrated action assignment converts these trajectories into actions. However, kinematic differences can leave residual pose errors, creating viewpoint mismatches between human demonstrations and robot observations. We therefore introduce Retargeting-Error-Aware Masking (REAM), which uses these errors during training to reduce reliance on mismatched views at deployment. Finally, because estimated actions cannot fully capture contact-rich interactions, we collect on-policy human corrections through DAgger to improve performance. We evaluate Pixel-UMI on five challenging tasks spanning articulated-object manipulation, deformable-object handling, constrained-space reaching, whole-body collision avoidance, and contact-rich bimanual coordination, achieving 87.0% average task progress. Video results are available at https://pixel-robot.com/pixel-umi