AI summaryⓘ
The authors point out that current video generation models make pretty pictures but don’t really understand how things move over time according to physical laws. They propose a new method called Latent Dynamics Reasoning (LDR), which models motion by explicitly calculating how objects change step-by-step, focusing only on the complex parts of movement while handling simpler movement with math formulas. Tested on a physics-based video benchmark, LDR predicts motions much better, especially when facing new situations it wasn't trained on, and it does so using fewer resources and faster speed than previous methods. This work is the first to show a video model that can extend what it learned about motion to completely new scenarios.
video diffusion modelslatent spacekinematic integrationdynamics modelingout-of-distribution generalizationphysics simulationwhite-box benchmarkmotion predictionresidual regressionmodel extrapolation
Authors
Haodong Li, Shaoteng Liu, Tianyu Wang, Chongjian Ge, Sihui Ji, Jiahan Zhang, Xin Lin, Haolin Lu, Zhe Lin, Manmohan Chandraker
Abstract
The world evolves following its dynamics, i.e., its laws of motion. However, leading video diffusion models largely fit the pixels without modeling how the pixels transit over time. Thus, they render visually plausible frames but may not accurately obey the laws. To capture the dynamics purely from pixels, we introduce Latent Dynamics Reasoning (LDR). LDR casts the latent transition as an explicit kinematic integration, where the lower-order dynamics are integrated numerically and the model regresses only the third- and higher-order residual that drives the rollout. For this integration to extrapolate better, LDR runs it on a structured latent rather than dense convolutional features. Following PhyWorld, we validate LDR on a controlled white-box physics benchmark spanning five tasks (uniform motion, parabola, collision, bouncing, looming), focusing on out-of-distribution scenarios that reveal whether a model has truly learned the underlying dynamics. LDR extrapolates the learned dynamics far better: the gap between its in- and out-of-distribution error is over 20$\times$ smaller than the video diffusion baseline's, under both single- and joint-task training at 256$^2$ resolution, while using 26$\times$ fewer parameters and running 143$\times$ faster. LDR can even generalize under severe shift: for example, trained only on red balls moving left-to-right, it correctly predicts the motion of a blue square moving right-to-left. To our knowledge, this is the first video world model that extrapolates learned dynamics beyond its training distribution. Project page: https://lat-dyn-reason.github.io/