Video diffusion models’ attention flaws cause physics errors in motion generation
Why Do Video Diffusion Models Violate Physics? Unveiling the Flaws in Attention Mechanisms
Computer Vision and Pattern RecognitionMachine Learning
Summary
Video diffusion models can create impressive videos but often make mistakes that break real-world physics, like impossible movement. The authors studied how these models plan motion during early stages and found that a part called Rotary Position Embedding (RoPE) causes the model to fix on wrong positions too early. This stops the model from creating realistic movement paths across frames. They propose a simple fix to adjust how RoPE works during video creation, which helps the model produce more physically believable motions without extra training.
What this means in practice
- •For video generation developers: Improve video diffusion software to create more physically plausible motion without needing extra data or complex priors.
- •For robotics simulation teams: Enhance synthetic video datasets used for training robots to better reflect real-world physics in motion.
Authors
Yueyan Li, Haibo Wang, Caixia Yuan, Xiaojie Wang
Abstract
Despite impressive visual quality, state-of-the-art video diffusion models often generate content that violates real-world physical laws. While existing solutions rely on external priors or specialized data, we investigate the root cause by exploring the internal mechanisms of these models. Specifically, we present the first interpretability study on the ''motion planning'' process of text-to-video diffusion models, revealing how motion trajectories form during early denoising stages. Building upon the ''first shape, then details'' finding, we combine cross-attention trajectory patterns with causal head contributions to identify a specific subset of attention heads driving motion planning. Further, our self-attention analysis shows that Rotary Position Embedding (RoPE) induces excessive spatial attention decay. This causes early candidate regions to prematurely lock into physically implausible positions, suppressing reasonable trajectories in adjacent frames and triggering generation failure modes. To address this fundamental flaw, we propose a lightweight architectural modification that scales the frequency of RoPE across different denoising steps. This strategy reduces excessive attention decay, helping the model explore better candidate regions to establish coherent physical motion. Finally, training-free and training-based experiments confirm the effectiveness of our approach in enhancing the physical commonsense of generated videos.