Human videos improve robot manipulation without matching motions
Cross-Embodiment Causal Transfer from Human Videos to Robot Manipulation
RoboticsArtificial Intelligence
Summary
Teaching robots to manipulate objects usually needs a lot of real robot practice, which is expensive and slow. The authors studied how watching human videos can help robots learn tasks even though human and robot bodies and moves are very different. They created a method that picks out important task details from human videos and uses robot examples to help robots act correctly without copying human movements exactly. Their experiments show that robots perform better on manipulation tasks using this approach.
What this means in practice
- •For robot developers: Improve robot task success rates by integrating human video knowledge without requiring precise human-robot motion matching.
- •For automation integrators: Use diverse human task videos to reduce costs and time in training robots for new manipulation tasks in varied settings.
Authors
Xingyu Zhang, Yingchao Li, Peizheng Guo, Jie Hu, Changwen Zheng, Wenwen Qiang
Abstract
Scaling robot manipulation models requires large amounts of interaction data, yet collecting diverse real-robot experience remains costly. Human manipulation videos, in contrast, provide abundant and diverse task experience that could potentially benefit robot learning. However, differences in morphology, action spaces, and control mechanisms make human motions unsuitable as direct supervision for robot actions, raising a scientific question: what knowledge in human videos should be transferred across embodiments to robot manipulation? To address this question, we analyze subtask transfer through the lens of causal beams, focusing on which contextual conditions and interaction dependencies should be retained to support local task effects. Based on this perspective, we propose Causal-Beam-Guided Subtask Transfer (CBST). CBST learns manipulation-related representations from human videos and grounds them in task-level semantic variables using robot demonstrations. Within a predefined manipulation model, causal-beam constraints guide contextual information selection by requiring retained assignments to sustain local mechanism outputs while preserving nontrivial responsiveness to retained inputs. The selected semantic information conditions a world-action model, allowing the robot to generate actions from its own observations without requiring human-robot motion correspondence. Experiments on robot manipulation benchmarks show improvements in task success over the corresponding world-action model baselines.