Video diffusion models achieve accurate camera control and smooth motion
WorldForge V2: Eliciting Camera-Trajectory Alignment in Video Diffusion Models
Computer Vision and Pattern Recognition
Summary
Controlling camera movements realistically in generated videos is tricky because models often mix up camera motion with the objects moving in the scene. The authors created a method called WorldForge V2 that can guide video generation along a desired camera path without messing up how things look or move. Their approach uses smart techniques to separate camera movement from object motion and makes corrections to keep the video smooth and natural. It requires very little new training and works well even on new scenes.
What this means in practice
- •For visual effects teams: Produce realistic camera movements in computer-generated video scenes while preserving object motion and appearance without high computational costs.
- •For game developers: Generate dynamic and spatially consistent game video content with direct camera control using pretrained video diffusion models and minimal additional training.
Authors
Chenxi Song, Yanming Yang, Tong Zhao, Ruibo Li, Joey Tianyi Zhou, Chi Zhang
Abstract
Video diffusion models have learned rich 3D/4D priors, making them promising for scene generation and video re-rendering. However, their use in spatial tasks is limited by imprecise camera control, spatial and temporal inconsistencies, and entangled camera motion with scene dynamics. Existing approaches, such as task-specific fine-tuning and post-process warping, often introduce visual artifacts, generalize poorly, or incur high computational costs. We propose WorldForge V2 to achieve accurate camera control while preserving the generative capabilities of pretrained video models. We first develop a training-free guidance framework using warped references to iteratively refine generation along a target camera path. Optical-flow analysis identifies motion-related latent channels for selective guidance, helping separate motion control from appearance. A dual-path strategy further compares guided and unguided predictions to correct artifacts from imperfect references. Together, these components guide viewpoint changes while preserving scene appearance and object motion. To reduce refinement costs, v-cache approximates the correction using the preceding denoising prediction, avoiding an additional model evaluation. We further extend motion localization from latent channels to network blocks and introduce Trajectory LoRA, which concentrates adaptation on blocks most responsive to trajectory guidance. A two-stage training strategy teaches these adapters to denoise the states encountered during guided generation, using limited supervision effectively. This enables reliable camera control with very little training data, with a single training scene already supporting generalization to unseen scenes. Quantitative and qualitative evaluations across multiple datasets show that our method achieves state-of-the-art generation quality and trajectory accuracy in camera-controlled video generation.