RealWonder: Real-Time Physical Action-Conditioned Video Generation
2026-03-05 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial IntelligenceGraphics
AI summaryⓘ
The authors present RealWonder, a system that creates videos showing how actions like pushes or robot moves change a scene, starting from just one image. They use physics simulations to turn these actions into visual clues that a video generator can understand, instead of trying to directly predict changes from the actions. Their system works fast enough to let people interact with different materials and objects in real time. This approach helps computers better understand how things move and react in 3D scenes.
video generationphysics simulation3D reconstructionoptical flowdiffusion modelsrobotic manipulationreal-time systemsdeformable bodiesgranular materials
Authors
Wei Liu, Ziyu Chen, Zizhang Li, Yue Wang, Hong-Xing Yu, Jiajun Wu
Abstract
Current video generation models cannot simulate physical consequences of 3D actions like forces and robotic manipulations, as they lack structural understanding of how actions affect 3D scenes. We present RealWonder, the first real-time system for action-conditioned video generation from a single image. Our key insight is using physics simulation as an intermediate bridge: instead of directly encoding continuous actions, we translate them through physics simulation into visual representations (optical flow and RGB) that video models can process. RealWonder integrates three components: 3D reconstruction from single images, physics simulation, and a distilled video generator requiring only 4 diffusion steps. Our system achieves 13.2 FPS at 480x832 resolution, enabling interactive exploration of forces, robot actions, and camera controls on rigid objects, deformable bodies, fluids, and granular materials. We envision RealWonder opens new opportunities to apply video models in immersive experiences, AR/VR, and robot learning. Our code and model weights are publicly available in our project website: https://liuwei283.github.io/RealWonder/