Video diffusion models show localized physical information during denoising
Does Physics Live in the Activations? Localizing Physical Quantities in Video Diffusion Models
Computer Vision and Pattern RecognitionMachine Learning
Summary
Video generation models often struggle to understand physical rules behind motion. This paper looks inside one such model and finds that, during the process of cleaning up noisy video data, the model builds a clear and accurate representation of physical quantities like object motion and forces. These physical details are especially present in parts of the model focused on objects themselves and can even be detected from single frames. The authors also show that this physical understanding is not just memorized but can generalize somewhat to new scenes.
What this means in practice
- •For video generation developers: Improve video synthesis quality by integrating localized physical information extracted from model activations to enhance realistic motion effects.
- •For robotics simulation teams: Use decoded physical quantities from video models as additional feedback for validating and refining physics-based simulators in robotics environments.
Authors
Jonas Kneifl, Jakub Skalski, Bartłomiej Twardowski, Kamil Deja
Abstract
Video generation models produce strikingly realistic sequences and are increasingly proposed as world models, yet recent benchmarks reveal pronounced deficits in their physical reasoning. This raises the question of whether these models internalize physical principles or merely reproduce familiar motion patterns. We address this by probing internal representations of video Diffusion Transformers (DiTs) for simulator-derived ground-truth physical quantities spanning kinematic motion and rigid-body dynamics under gravity and contact. We find that these quantities are linearly decodable with high accuracy early in the denoising process, substantially outperforming a baseline decoded directly from the model's own noised latents, indicating that the relevant physical information is actively constructed during denoising rather than already present in the input. Additionally, we show that activations at on-object tokens carry the relevant physical information and that quantities defined over multiple frames are readable from single latent frames. Hence, information is sharply localized within the token sequence and is computed globally but stored locally. The probes further show partial extrapolation, transferring to scene variations and object configurations outside their training regime, so what they read is not simply a correlate of the scenes they were fit on. When fitted directly in the full-resolution activation space, the probing directions can serve as steering vectors to change the model's output.