Physics aware optical flow improves video motion realism and control
PhysFlow: Physics-Aware Optical Flow for Motion Controllable Video Generation
Computer Vision and Pattern Recognition
Summary
Generating videos that look real and move according to physical rules is difficult for computers. The authors propose PhysFlow, a new method that first predicts how objects move using a physics-aware model, then uses this motion to create the final video images. They also built a large dataset of videos simulated with known physical properties to help train the system. This approach helps produce videos where motion looks more natural and believable while keeping the visuals clear.
video generationoptical flowphysical realismmotion modelingappearance synthesisphysics simulationattention mechanismmachine learningvideo datasetmaterial properties
Authors
Cong Wang, Hanxin Zhu, Yonglin Tian, Jiayi Luo, Ruiqi Song, Boyi Sun, Long Chen, Zhibo Chen
Abstract
Video generation models have recently attracted substantial attention for their ability to generate visually compelling videos, yet ensuring physically consistent and plausible dynamics still remains a fundamental challenge, driving a growing line of research on physical realism in video generation. To address this challenge, motivated by the fact that physical regularities are primarily encoded in motion patterns, we propose PhysFlow, a novel two-stage framework for improving the physical plausibility of generated videos by decomposing video generation into motion-aware optical flow generation followed by motion-conditioned appearance synthesis. Specifically, PhysFlow consists of a physics-aware optical-flow video generator called PA-Flow and a flow-guided video generator called FlowRender. During the first stage, PA-Flow employs a physics-aware attention module to model how motion attributes and material properties influence global motion and local deformation, respectively, and generates an optical flow video as an explicit representation of motion. In the second stage, FlowRender leverages the decoupled motion representation as guidance to synthesize realistic textures and appearances, ultimately producing the final physically plausible video. To further support model training with explicit physical supervision, we construct PhysVideo, a physics-based video dataset generated with a physics engine and 3D-GS rendering, containing 10K foreground objects and 50K realistic video sequences with annotations of motion and material properties. Extensive experiments demonstrate that our proposed PhysFlow generates videos with superior physical plausibility while maintaining high visual fidelity compared with existing methods.