Physics based video generation enables fine control of object motions

PhysStream: Streaming Physics-Grounded Video Generation with Structured Scene Memory and Fine-Grained Motion Control

Computer Vision and Pattern RecognitionArtificial IntelligenceGraphics

Summary

Videos showing moving objects are often hard to control with great detail, especially when trying to follow real physical rules during movement. PhysStream is a new method that builds videos step-by-step by remembering where objects are and how they move, using signals related to speed and direction to guide the motion more naturally. The creators trained their method in two parts to make videos that follow physical laws better and let users change motion while the video is still being made. This method works especially well for simple tabletop scenes with multiple solid objects, and people who watched preferred its videos over others most of the time.

What this means in practice

  • For game developers: Create interactive video scenes where game objects move realistically and can be adjusted during gameplay using physical motion signals.$Commercial implications: Enables production of dynamic game content with controllable physics-based animations, enhancing player experience and reducing animation design time.
  • For visual effects artists: Generate videos of moving objects on tabletops with precise physical consistency and adjust their motion details interactively during editing.

Authors

Chuhao Chen, Peter Wonka, Chaoyang Wang, Chen Wang, Qiao Feng, Sergey Tulyakov, Lingjie Liu

Abstract

Interactive control for video generation is moving from coarse prompts toward fine-grained, physically meaningful manipulation of dynamic scenes. Yet existing controllable methods either require the full control schedule before generation starts, or use pixel-space signals that dictate object positions rather than physical dynamics. To address these limitations, we propose PhysStream, an autoregressive model for physics-grounded image-to-video synthesis that incorporates structured scene memory---positional maps and object tracking maps derived online from previously generated frames---and supports fine-grained motion control via sparse velocity-increment signals that encode physical quantities, letting the model learn the underlying dynamics. We train our model in two stages: a bidirectional model is first finetuned with motion-control conditioning, then a causal autoregressive model is trained with additional structured scene memory, further improving physical consistency. PhysStream enables interactive, mid-generation control over multi-object tabletop rigid-body scenes---a capability not supported by prior methods---reducing motion distribution distance (FVMD) by 33% and trajectory error by 12% over the strongest baselines on synthetic benchmarks, and is preferred by human evaluators in over 85% of in-the-wild comparisons. Please check our website for more details: https://czzzzh.github.io/PhysStream
View PDFOpen arXiv