Robotic manipulation improved by geometry-aware multi-view action modeling
MVG-WAM: Multiple View Geometry-Aware World-Action Modeling for Robotic Manipulation
Robotics
Summary
Robots need to understand their surroundings from multiple camera views to manipulate objects accurately. This paper introduces a new model that better connects images from different cameras by using the geometry of the scene, making it easier for robots to predict actions. The model uses information about the camera setup and depth to build a clear picture of the 3D world without heavy processing during action steps. Tests show it works well in both simulated environments and real-world robot tasks.
What this means in practice
- •For robotic system developers: Build robots that can better interpret multiple camera views to improve manipulation accuracy in complex environments.
- •For augmented reality hardware teams: Enhance AR devices by integrating geometry-aware multi-view models to improve spatial understanding from multiple cameras.
Authors
Wenbo Chen, Tianfu Li, Haoxuan Xu, Zhihao Cao, Zhenghan Chen, Zhengming Zhu, Zizhou Luo, Guosheng Yang, Yuan Liu, Lujia Wang, Wen Chen, Haoang Li
Abstract
World-Action Models (WAMs) couple visual dynamics with action prediction, bringing the rich priors of pretrained video models to robotic manipulation. However, their multi-view interfaces typically tile images or concatenate tokens, leaving the geometric relationships among synchronized cameras implicit. This makes it harder to connect global scene context with the local geometry required for interaction. We introduce the Multi-View Geometry-Aware World-Action Model (MVG-WAM), which organizes these observations as related projections of one physical world rather than separate images on a canvas. Our model combines an epipolar-constrained global state with view-indexed geometric states jointly inferred from synchronized observations. Camera-aware routing supplies each video region with its corresponding geometric context and the shared global state, explicitly structuring the representation used for action prediction. We further ground the geometry-aware representation in metric scale through multi-horizon future-depth supervision, without requiring depth decoding during action rollout. MVG-WAM achieves average success rates of 99.1% on LIBERO and 92.07% on RoboTwin 2.0, demonstrating competitive performance across both benchmarks. Real-world experiments on Cobot Magic further demonstrate a 91.3% success rate across 150 trials spanning three manipulation tasks.