Robot manipulation improved by new visual feedback motion planning
Vector-Quantized World Model: Latent Feedback Motion Planning for Robot Manipulation
Robotics
Summary
Long and complex robot tasks are hard because small errors add up and planning can be too complicated. The authors created a system called Vector-Quantized World Model that plans robot actions using a special way to imagine steps ahead and then adjusts motions visually to fix mistakes as they happen. They tested it by having a robot solve the Tower of Hanoi puzzle and it succeeded every time, even when it learned only from short examples. This method also did better than other recent models on various robotic tests in simulations.
What this means in practice
- •For robotics engineers: Plan and execute long sequences of robot arm movements reliably using a combined discrete search and continuous visual feedback control system.
- •For automated logistics teams: Enable robotic manipulators to adaptively complete complex stacking or assembly tasks that require multi-step planning beyond demonstrated examples.
Authors
Xuezhou Xu, Chung-Pang Wang, Zhenzhen Li, Benjamin Riviere
Abstract
Visual world models have shown strong potential for robot planning, yet long-horizon manipulation remains challenging due to compounding prediction errors, large search complexity, and latent plans that the low-level controller cannot reliably reach. In this work, we introduce Vector-Quantized World Model (VQ-WM), a latent feedback motion planning framework that integrates discrete long-horizon search with continuous visual feedback control: a discrete latent predictor models transitions between vector-quantized latent states, allowing the planner to look far ahead and find a shortest path to the goal through discrete search and a continuous learned controller sequentially reaches the planned states through visual feedback, correcting deviations to mitigate compounding errors. Whereas most vector quantization uses Euclidean distance, we use a novel estimator of the first-passage-time as a quasi-metric, which aligns the planning abstraction with the controller's capabilities. We evaluate on the physical robot manipulation of the Tower of Hanoi puzzle: whereas existing state-of-the-art foundation models (VLAs, WAMs) only succeed when the solution trace in their training dataset, our method achieves 100\% success rate with only short-play data, demonstrate its ability to compose long-horizon solutions beyond the short demonstrated trajectories and execute them reliably. Our solution also outperforms strongest world-models by 13.3--85.3 percentage points on simulation benchmarks spanning manipulation and locomotion, demonstrating consistent progress and new capabilities.