Hallucination-aware world model improves general robot manipulation success
HaWMPO: Hallucination-Aware World Model-based Policy Optimization for Generalist Robot Policy
Robotics
Summary
Robots that can do many tasks often struggle with long and complex sequences of actions. The authors found that training robots using imagined experiences called world models can help, but these imaginations sometimes have mistakes, called hallucinations, that confuse the robot. They created a new method called HaWMPO that watches out for these hallucinations and lowers their influence on learning. This approach improved robot success rates significantly in tests and real-world tasks.
What this means in practice
- •For robotics engineers: Enhance training of robot manipulation policies by reducing errors from simulated training data in long task sequences.
- •For automation system designers: Improve deployment reliability of generalist robot systems performing multi-step real-world tasks by using hallucination-aware model training.
Authors
Zengjue Chen, Peidong Liu, Jiawei Li, Qi Wang
Abstract
Generalist robot policies have demonstrated strong generalization across robotic manipulation tasks, yet their success rates remain limited in com- plex long-horizon scenarios. Recent methods improve Visual-Language-Action (VLA) policies through online reinforcement learning on real robots, but such training relies on costly physical interactions, suffers from low sample efficiency, and may introduce hardware and safety risks. World models offer a promising alternative by enabling policy optimization with imagined rollouts. However, long-horizon rollouts generated by world models often suffer from prediction hal- lucinations, producing biased state transitions that can mislead policy learning. To address this issue, we propose Hallucination-aware World Model-based Pol- icy Optimization (HaWMPO), a closed-loop reinforcement learning pipeline for VLA policy post-training with world models. Specifically, HaWMPO introduces an action-conditioned hallucination-aware model to estimate the reliability of gen- erated image sequences, and incorporates hallucination scores into group relative policy optimization through a Reward-Soft mechanism, suppressing unreliable ac- tion chunks during training. On the LIBERO benchmark, HaWMPO achieves the best average success rate, with gains of 15.0% over the base model and 2.8% over the strongest baseline; real-world experiments on a G1 robot further validate its effectiveness, raising the average success rate on two manipulation tasks from 67.5% to 80.0%.