Robot learns to match actions with visual task predictions
Achieve What You Imagined: Learning to Align Actions with Visual Plans
RoboticsArtificial IntelligenceMachine Learning
Summary
Robots can imagine future scenes and actions to complete a task, but these imaginations don’t always line up. The authors found that robots can predict good-looking task outcomes before figuring out the right actions to achieve them. They use the imagined outcome as a goal and improve the robot’s action plan by comparing different predictions to find consistency. This approach improved the robot’s success rate in four real-world arm tasks without needing extra training or testing on the robot.
What this means in practice
- •For robotics engineers: Improve robot arm task success rates by refining action plans using cross-model visual consistency without extra robot trials.
- •For automation integrators: Deploy enhanced visual plan alignment methods for robotic manipulators in manufacturing to boost reliability in task completion.
Authors
Yuheng Qiao, Ziran Wei, Xiaohan Wang, Daqiang Guo, Yichen Luo, Zhibo Pang, Peng Zhou, Sichao Liu
Abstract
World-action models can jointly predict future visual observations and robot actions. However, discrepancies may exist between their visual predictions and the consequences implied by generated actions. We observe that WAMs can often generate visually plausible task-completion outcomes before producing action sequences that reliably achieve them. Consequently, we treat the WAM-generated visual prediction as a goal-conditioned visual proposal rather than a directly executable plan. We use a frozen action-conditioned world model to predict action-conditioned consequences and construct feedback based on consistency between the two future predictions and alignment with the terminal goal. Leveraging this feedback, we employ Flow Policy Optimization (FPO) to optimize the action head of the WAM. This framework avoids online robot interaction and additional training of task-specific reward models. Across four real-world UR5 manipulation tasks, our method increases the mean success rate from 43.4% to 75.1%, compared with 61.4% for $π_{0.5}$. These results show that cross-model prediction discrepancy can provide useful feedback for improving robot policies under the evaluated manipulation tasks. Website: https://imagine-to-achieve.github.io/