StereoEngine creates stereo robot data without stereo cameras
StereoEngine: Scaling Stereo Bimanual Robot Data without Stereo Cameras
RoboticsComputer Vision and Pattern RecognitionMachine Learning
Summary
Robotic systems that use stereo vision to see better often need large sets of stereo camera videos, which are hard to get. The authors developed StereoEngine, which can make stereo video pairs from ordinary single-camera videos by estimating depth and filling in missing parts, or by creating fully synthetic stereo scenes in simulation. Their approach helps robots learn better ways to use two hands, even without real stereo recordings. This work improves training data for robot manipulation without relying on expensive stereo cameras.
What this means in practice
- •For robotic system developers: Generate realistic stereo video data from monocular videos to improve training of bimanual manipulation policies without needing stereo cameras.
- •For virtual training platform builders: Create photorealistic stereo pairs in simulation for synthetic datasets that help train manipulation robots more effectively.
Authors
Jason Chen, I-Chun Arthur Liu, Gaurav Sukhatme, Daniel Seita
Abstract
Stereo vision is increasingly used for robotic manipulation, with a growing body of work incorporating it into Vision-Language-Action (VLA) models. However, the scarcity of large-scale, real-world stereo datasets remains a fundamental bottleneck for scaling such policies. Recent generative monocular-to-stereo methods target settings such as VR content and autonomous driving. No prior work produces realistic synthetic stereo data for robotic manipulation, where stereo supervision has come predominantly from simulation. We propose StereoEngine, which introduces two complementary methods to synthesize stereo data for robotic manipulation. In the first, we warp a monocular (left) video into the right view using estimated depth and inpaint the disoccluded regions with a video diffusion model, lifting monocular demonstrations into stereo pairs without training on real stereo. In the second, we render a stereo pair in simulation and jointly generate both photorealistic views, producing consistent left-right data without any real video as input. StereoEngine outperforms existing baselines in generated stereo quality, and using its synthetic data for policy mid-training or task-specific fine-tuning substantially improves downstream bimanual manipulation. Project website: https://stereoengine.github.io