World action model improves robot spatial prediction for manipulation tasks
XGenAct: Geometry-Enhanced World Action Models through Cross-Task Generation
RoboticsComputer Vision and Pattern RecognitionMachine Learning
Summary
Robots need to understand their surroundings in detail to handle objects well. The authors created XGenAct, a model that helps robots predict future actions and what they will see, including depth and object roles, all at once. This combined approach improves the robot's success in completing tasks compared to methods that only use normal camera images. It works by training one model to predict different types of information as video, rather than treating each type separately.
What this means in practice
- •For robotics engineers: Improve robot control systems by combining spatial perception tasks into one model for better manipulation accuracy.
- •For automation system integrators: Develop robots that better predict and react to object roles and geometry during manipulation tasks in industrial settings.$Commercial implications: Enables advanced industrial robots with improved spatial understanding for tasks like assembly, boosting automation product value.
Authors
Tingting Du, Ziyao Wang, Guoheng Sun, Ang Li
Abstract
World action models (WAMs) have advanced robot control by predicting how observations and actions evolve over time. Despite this progress, RGB and action based future prediction does not explicitly address the spatial understanding needed for robot manipulation. Existing efforts often add a limited set of spatial prediction tasks through specialized heads or branches, leaving both the range of spatial supervision and the model architecture fragmented. We introduce XGenAct, a world action model that represents RGB observations, robot actions, metric depth, surface normals, and functional role segmentation as RGB videos through deterministic codecs. By sampling perception and action tasks during training, XGenAct uses one video diffusion transformer and one objective to learn temporal prediction across these spaces without modality specific learned heads. On held out RLBench tasks, structured perception training improves average closed loop success over RGB only training, and XGenAct achieves 52% success in the five task external comparison, versus 26% for the strongest evaluated baselines. It also predicts future depth and segmentation more accurately than the evaluated pipelines that generate RGB first and then apply a frozen perception expert.