Visual embodiment structure improves robot control and task learning
Rethinking Visual Embodiment Dependence in Visuomotor Policies
Robotics
Summary
When robots learn tasks by watching scenes and their own parts, they might rely too much on how their body looks instead of focusing on the task. The authors show that instead of removing the robot’s visual body, it’s better to represent it in a standard way that keeps important control details but hides robot-specific shapes. This helps robots learn better from tasks done by humans and handle new robot designs more robustly. They also add training tricks to avoid robots cheating by just looking at their configuration rather than the task progress.
What this means in practice
- •For robotics engineers: Enable robots to better learn tasks demonstrated by humans without needing robot-specific training data by using canonicalized visual body representations.
- •For robot software developers: Improve robot control software robustness across different robot configurations by using embodiment canonicalization and configuration-decoupling training augmentations.
Authors
Hongjie Fang, Yuxuan Lu, Chenxi Wang, Haoxiang Qin, Shirun Tang, Zihao He, Shangning Xia, Jingjing Chen, Wanxi Liu, Shiquan Wang, Cewu Lu
Abstract
Visuomotor policies observe both the task scene and the acting embodiment, allowing embodiment-specific visual cues to influence action prediction. We study this phenomenon as visual embodiment dependence (VED) and show, through cue-conflict interventions across representative policies, that visible robot configuration can become a shortcut to task progress. Rather than eliminating VED, we argue that it should be structured around embodiment information that supports control and generalization. We realize this through embodiment canonicalization in 3D point clouds, replacing the original embodiment with a canonical end-effector representation (CER) that preserves control-relevant geometry while abstracting embodiment-specific morphology. Its editable form further enables configuration-decorrelation augmentation for unfamiliar robot configurations. Experiments show that embodiment canonicalization substantially improves human-to-robot policy transfer without robot demonstrations, while simply removing the embodiment is insufficient without preserving control-relevant geometry. We further find that CER itself can become a configuration shortcut when robot configuration becomes decoupled from task progress; configuration-decorrelation augmentation mitigates this failure mode and restores robust recovery without sacrificing performance on seen configurations. Together, these results show that robust visuomotor learning benefits from structuring, rather than removing, visual embodiment information. Project website: https://tonyfang.net/ved