Video generation improves physical realism with hierarchical alignment
MiMaAlign: Hierarchical Micro-Macro Alignment for Physically Plausible Video Generation
Computer Vision and Pattern Recognition
Summary
Generating videos from text is getting better, but sometimes the results show impossible actions or events that don’t make sense physically. The authors designed a method called MiMaAlign that teaches the video generator to understand movement and interactions better by using two types of guidance: one that focuses on smooth continuous motions and another that looks at higher-level events and object relationships. This approach helps the video generator produce more realistic and physically plausible videos, improving performance on tests measuring common sense about physical events.
What this means in practice
- •For video content creators: Generate videos with more realistic and physically consistent motion and event interactions based on text prompts.
- •For augmented reality developers: Create virtual event sequences that adhere to physical commonsense, improving plausibility in mixed reality experiences.
Authors
Yansong Liu, Litong Gong, Weijie Li, Tiezheng Ge, Ming Zeng
Abstract
Text-to-video (T2V) diffusion models have made remarkable progress in generating high-fidelity visuals, yet their outputs still often exhibit physically implausible phenomena, including discontinuous dynamics and implausible event interactions. Existing representation alignment methods mainly transfer generic video features or short-range motion cues, treating physical plausibility as a single-level representation problem. To address this issue, we propose MiMaAlign, a hierarchical micro-macro alignment framework that transfers complementary dynamics-oriented and event-oriented knowledge into different abstraction depths of a pretrained video diffusion model. Specifically, micro-level continuous dynamics alignment imposes constraints by aligning diffusion features with a continuous-time motion representation learned by a Neural Ordinary Differential Equation (Neural ODE), while macro-level event-commonsense alignment provides representations of object interactions and event semantics through an instruction-prompted vision-language model (VLM). With training-only auxiliary objectives, we align the complementary supervision from the two teachers with shallower and deeper generator features, respectively, enabling the generator to internalize the corresponding physical priors. Empirical evaluations show that, on CogVideoX-2B, MiMaAlign achieves a relative improvement of up to 21.0% in the Physical Commonsense (PC) score on the VideoPhy benchmark over the baseline, and also delivers substantial gains on the highly challenging VideoPhy2 benchmark, while maintaining general video quality on VBench and being supported by blind user comparisons.