Robot model predicts actions using views from camera and wrist sensors
DUET-DINO: Simultaneous Cross-View World Modeling for Latent Planning in Robot Manipulation
RoboticsComputer Vision and Pattern Recognition
Summary
Robots often find it hard to predict where and how to move their arms precisely, especially in complex ways. The authors created a system called DUET-DINO that learns to predict future robot actions by combining what a camera looking at the whole scene sees with what a wrist-mounted camera sees. This helps the robot plan better movements involving detailed positions and rotations. Their system worked well in different tasks like reaching, angled reaching, and picking up objects, performing better than other methods. DUET-DINO also stays reliable even when the robot’s visual input changes, thanks to training on diverse data.
latent world modelsrobot manipulation7-DoF controlcross-view conditioningvisual representationgoal-conditioned planningrobot graspingvisual distribution shifts
Authors
Nisarga Nilavadi, Ralf Römer, Moritz Reuss, Michael Krawez, Tobias Jülg, Angela P. Schoellig, Rudolf Lioutikov, Wolfram Burgard
Abstract
Action-conditioned latent world models predict future visual representations, enabling zero-shot goal-conditioned robot planning and control. However, their predictions for fine-grained spatial and rotational actions are unreliable for full 7-DoF end-effector control. To address this gap, we introduce DUET-DINO, a simultaneous cross-view latent world model that jointly learns action-conditioned predictions from static side- and wrist-camera observations through cross-view conditioning. By exploiting complementary global scene and gripper-centric information, DUET-DINO enables latent planning over the full 7-DoF action space. Across spatially diverse reach, orientation-intensive angled-reach, and multi-goal grasp-and-lift tasks, DUET-DINO consistently outperforms single-view and independent dual-view baselines, achieving 92% success on reach, 72.5% on angled-reach, and 60.0% on lift tasks. DUET-DINO is trained from scratch on DROID and RoboArena datasets and generalizes robustly under visual distribution shifts. We further show that while V-JEPA 2 wrist-view predictions underestimate visual dynamics induced by fine-grained actions, DINOv3 predictions better capture action-conditioned scene changes, leading to stronger downstream planning. The code and model checkpoints will be open-sourced. Project page: https://utn-air.github.io/DUET-DINO