Video model enhanced to create realistic fluid motion using physics data
FluidGen: Physics-Aware Fluid Video Generation via Motion-Guided Post-Training
Computer Vision and Pattern RecognitionGraphics
Summary
Generating videos of fluids like smoke and fire is hard because typical video models don’t follow real fluid motion rules, making them look fake. The authors combined simulated physics data with real fluid videos to teach a video model to predict velocity fields along with the visuals. This helps the model create more realistic and physically accurate fluid movements. Their approach also uses these velocity predictions to guide improvements, resulting in better fluid animations than previous methods.
What this means in practice
- •For visual effects artists: Generate fluid animations with realistic physics-based motion from input videos or scenes without detailed 3D setup.$Commercial implications: Enables fluid effects creation for film and game studios without complex simulations, making production faster and more flexible.
- •For video game developers: Create believable fluid dynamics in game environments by augmenting existing video generation models with physical motion data.
Authors
Yang Zheng, Zizhang Li, Boyang Deng, Jan Ackermann, Lei Shu, Yinghao Xu, Jiajun Wu, Gordon Wetzstein, Leonidas Guibas
Abstract
Video generation models produce visually compelling motion but systematically fail on fluid phenomena, including smoke, fire, and explosion debris, where outputs lack coherent vortical structure and collapse to bland dynamics that violate the physics of real fluids. Classical solvers achieve high physical fidelity but require detailed 3D scenes and carefully tuned parameters, making them impractical for in-the-wild imagery and video generation. We address this gap by injecting a data-driven fluid prior into a pre-trained video model, trained on a mixture of high-fidelity Houdini simulation data with paired velocity-field ground truth and curated in-the-wild fluid video data for appearance diversity. The core of our approach is a dual stream architecture that predicts per-frame velocity fields alongside RGB, supervised by simulation ground truth. This auxiliary signal regularizes generation toward physically consistent motion and exposes an interpretable physical quantity that we further leverage as a reward for velocity-aware direct preference optimization. Our model produces substantially more realistic fluid dynamics sequences than strong video baselines and optimization-based methods. Project page: https://y-zheng18.github.io/FluidGen/