Duet-DINO improves robot grasping using two camera views
DUET-DINO: Simultaneous Cross-View World Modeling for Latent Planning in Robot Manipulation
RoboticsComputer Vision and Pattern Recognition
Summary
Robots often plan their movements by predicting how the world will look after each action, but this can be tricky when the robot’s hand moves precisely in all directions. The authors introduce Duet-DINO, a system that learns to predict from two camera views at once — one from the side and one from the robot’s wrist — so it can better understand how actions will change what it sees. This helps the robot plan more accurate movements with its full range of motion, leading to improved success in reaching and grasping tasks. Their method works well even when the visual environment changes and outperforms approaches using only one camera view.
What this means in practice
- •For robotics engineering teams: Plan robot arm actions that require precise positioning and orientation by using dual camera views for better action prediction.
- •For automated warehouse operators: Improve robot picking and lifting tasks in cluttered environments by enabling more reliable grasp-and-lift planning from multiple visual perspectives.
Authors
Nisarga Nilavadi, Ralf Römer, Moritz Reuss, Michael Krawez, Tobias Jülg, Angela P. Schoellig, Rudolf Lioutikov, Wolfram Burgard
Abstract
Action-conditioned latent world models predict future visual representations, enabling zero-shot goal-conditioned robot planning and control. However, their predictions for fine-grained spatial and rotational actions are unreliable for full 7-DoF end-effector control. To address this gap, we introduce DUET-DINO, a simultaneous cross-view latent world model that jointly learns action-conditioned predictions from static side- and wrist-camera observations through cross-view conditioning. By exploiting complementary global scene and gripper-centric information, DUET-DINO enables latent planning over the full 7-DoF action space. Across spatially diverse reach, orientation-intensive angled-reach, and multi-goal grasp-and-lift tasks, DUET-DINO consistently outperforms single-view and independent dual-view baselines, achieving 92% success on reach, 72.5% on angled-reach, and 60.0% on lift tasks. DUET-DINO is trained from scratch on DROID and RoboArena datasets and generalizes robustly under visual distribution shifts. We further show that while V-JEPA 2 wrist-view predictions underestimate visual dynamics induced by fine-grained actions, DINOv3 predictions better capture action-conditioned scene changes, leading to stronger downstream planning. The code and model checkpoints will be open-sourced. Project page: https://utn-air.github.io/DUET-DINO