Do Robotic World Models Really Follow Actions? Diagnosing and Aligning Action-Conditioned Generation for Policy Learning
2026-08-25 • Robotics
RoboticsComputer Vision and Pattern Recognition
AI summaryⓘ
The authors study action-conditioned world models, which predict what will happen when a robot takes certain actions. They find that while these models work well when following expert actions, they struggle with less typical, off-expert actions, often ignoring commands or producing unrealistic visual results. To fix this, they create WorldSync, a method that improves the models by training them on more varied actions, grounding video predictions in robot movements, and better matching predicted changes to actual outcomes. Tests show WorldSync leads to better predictions and helps robots learn more successful policies.
world modelsaction-conditioned predictionrobot dynamicspolicy evaluationoff-expert actionsvisual integritySE(3) trajectory alignmentsimulationpolicy improvementrobot learning
Authors
Sixiang Chen, Jiaming Liu, Jixian Wu, Yichen Guo, Tinghao Wang, Siyuan Qian, Hao Chen, Jiajun Cao, Jian Tang, Shanghang Zhang
Abstract
Action-conditioned world models are increasingly used as learned simulators for policy evaluation and improvement, yet their effectiveness rests on an unverified assumption: generated futures faithfully reflect arbitrary valid actions. Existing benchmarks are typically confined to expert demonstrations, leaving off-expert action following inadequately evaluated. To address this gap, we introduce WorldEcho, which probes action following over a broader action distribution using visual integrity and SE(3) trajectory alignment. Our diagnosis shows that current world models reasonably execute expert actions but struggle with diverse off-expert trajectories, either ignoring the commanded actions or producing visually invalid rollouts. We further propose WorldSync, which strengthens action following along three complementary axes: distributional coverage, representational grounding, and intervention-effect alignment. It broadens the training distribution over action consequences, grounds intermediate video representations in action-induced robot dynamics through an Action-Forcing Expert, and aligns predicted changes under action interventions with the corresponding changes in ground-truth futures. Experiments on RoboTwin benchmarks and real-robot tasks show that WorldSync improves WorldEcho metrics and serves as a more reliable simulator for iterative policy improvement, enabling policies to achieve higher success rates.