Video generation models improved for steady camera movement
CausalAlign: Causal Alignment of Video Generation Models for Consistent Ego-Motion
Computer Vision and Pattern Recognition
Summary
Videos created by AI models often have shaky or drifting camera movements that make them look less realistic over time. The authors studied how these models handle changes in viewpoint and found that the way they represent camera motion fades during the video creation process. They designed a method called CausalAlign that helps the models keep track of camera movements more accurately by linking their internal signals to a 3D model of motion between frames. This leads to smoother and more consistent videos with better geometry and motion, helping both video creation and robot tasks that rely on understanding movement.
What this means in practice
- •For video content creators: Generate AI videos with more stable and consistent camera movements for improved visual quality.
- •For robotics software developers: Enhance robot manipulation planning by using improved video models that keep consistent world views over time.
Authors
Kai Zhu, Yintianrun Zhang, Shuwei Jiang, Ruolin Zuo, Yixin Chen, Duomin Wang, Miao Liu, Yin Li, Daquan Zhou, Yiwu Zhong
Abstract
Diffusion-based video generation models (VGMs) can produce visually plausible videos, yet their camera ego-motion often drifts over long horizons. Camera ego-motion determines how the viewpoint changes between frames and is essential for maintaining consistent geometry and trajectories. To address this problem, we propose CausalAlign, a causal alignment framework with two components. First, we diagnose how precise ego motion is represented, used, and retained within VGMs using linear probes and causal interventions, revealing a rise-and-fall pattern during denoising. Second, we strengthen ego-motion state in VGM by aligning the representations at the identified layers to a 3D foundation model through delta-token space. This space encodes the changes between consecutive frames, allowing the alignment to focus on ego-motion dynamics rather than static scene content. With Wan2.2 fine-tuned on DL3DV and RealEstate10K, CausalAlign can largely improve motion accuracy, geometric consistency, and temporal smoothness, while preserving visual quality. For example, it improves geometric consistency by more than 20% relative to the base model. Moreover, with the fine-tuned VGM, CausalAlign improves the success rate of world action models (WAMs) by 10 points on mobile manipulation tasks of RoboCasa365 benchmark. Overall, CausalAlign improves ego-motion consistency in diffusion-based VGMs and yields noticeable gains in embodied models.