Dream.exe: Can Video Generation Models Dream Executable Robot Manipulation?
2026-06-03 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors created Dream.exe, a tool that tests how well video generation models understand real-world physics by turning their generated videos into robot actions. They check if a robot can follow the motions shown in the videos to complete tasks in a simulator, which visual tests alone can't measure. They tested 8 different models on various tasks and found that some models do represent physical laws well enough to guide robot movement. However, looking good visually doesn't always mean the robot can successfully do the task. This work helps reveal how video models understand physical reality beyond just images.
video generation modelsrobotic manipulationphysical lawsvideo-to-execution pipelinerobot trajectoriesphysics simulationmanipulation tasksvisual qualityexecution successgenerative priors
Authors
Rui Zhao, Kaiming Yang, Jifeng Zhu, Siyang Chen, Ziqi Wang, Weijia Wu, Kevin Qinghong Lin, Heng Wang, Mike Zheng Shou
Abstract
Video generation models have made impressive strides in synthesizing visually compelling content, yet their outputs remain confined to the virtual domain. A natural question follows: how well do these models reflect the physical world when their generated videos leave the screen and enter reality? We propose robotic manipulation as a concrete, measurable window onto this question: if a model has truly internalized physical laws, the motion it depicts should translate into executable robot behavior. We introduce Dream.exe, an evaluation framework that operationalizes this criterion through a video-to-execution pipeline. Given a scene image and a task description, Dream.exe synthesizes a manipulation video, converts the generated motion into robot trajectories, and executes them in a physics simulator, yielding a grounding signal that purely visual metrics cannot offer. Using this pipeline, we evaluate 8 models spanning frontier closed-source generators, open-source generators, and robot-specific models. Our benchmark covers 101 manually curated manipulation tasks at three levels of physical complexity, measured across visual quality, trajectory fidelity, and execution success. Encouragingly, several models achieve measurable execution success, suggesting that generative priors learned from internet-scale data already encode meaningful physical knowledge. Yet visual quality proves a poor predictor of executability, exposing a dimension of model capability that standard visual evaluations do not capture. Dream.exe will be open-sourced at https://github.com/showlab/Dream.exe.