MobileVISTA improves robot task performance across varied poses
MobileVISTA: Generative Data Augmentation for Pose Generalization in Mobile Manipulation
RoboticsComputer Vision and Pattern RecognitionMachine Learning
Summary
Robots that move around while handling objects struggle when their starting positions change even a little, making their vision and movements unfamiliar. The authors created MobileVISTA, a way to create new training data by changing both the robot's camera views and its actions to match different poses. This makes the robot's learned skills more reliable when its position at work varies. They tested this on different robot types and found it helps most when the robot’s camera moves with its body parts.
What this means in practice
- •For robotics engineers: Train manipulation policies that remain robust despite changes in robot base position using augmented demonstration data without extra hardware or new data collection.
- •For automation system developers: Improve reliability of mobile robots performing dexterous tasks in dynamic settings by using pose-aware data augmentation to handle variable camera views.
Authors
Suzannah Wistreich, Stephen Tian, Isabella Huang, Vitor Campagnolo Guizilini, Sergey Zakharov, Katherine Liu, Jiajun Wu
Abstract
Mobile manipulators such as humanoid robots are increasingly deployed in dynamic, unstructured environments to perform dexterous manipulation tasks. However, end-to-end manipulation policies trained to imitate demonstration data collected from a single robot pose are brittle: even centimeter-scale deviations in robot pose at deployment can drive ego-centric observations and end-effector trajectories out of the training distribution, leading to sharp drops in performance. We introduce MobileVISTA, a data generation framework that transforms demonstrations captured at canonical poses into diverse, pose-perturbed training data by jointly (1) augmenting egocentric visual observations and (2) retargeting actions to compensate for base pose changes. Unlike prior methods, which assume a camera rigidly mounted off the actuated chain or non-trivial articulated robot geometry largely out of frame, MobileVISTA targets compatibility with egocentric platforms (e.g., humanoids) where the camera is both influenced by and must observe the robot's kinematic chain as it moves. We study MobileVISTA in simulated tasks spanning humanoid and bimanual embodiments, and on a real Galaxea R1 Pro. We find policies trained on MobileVISTA-augmented data demonstrate improved robustness to previously out-of-distribution poses encountered at test time, without additional demonstration collection or a trained generative model. Additionally, we find MobileVISTA's benefit is largest on tested humanoids, where the camera rides the actuated chain and the robot fills much of the frame. Additional videos and appendix can be found on our website: https://mobilevista.github.io