Camera controlled video generation improves with new particle filtering method
PartiCam: Camera Controlled Video Generation with Reward Guidance
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Generating videos that follow a specific camera movement is hard for existing AI models. The authors propose PartiCam, a method that guides video creation without retraining large models by using particle filtering techniques. This approach helps videos stick more closely to the planned camera path while keeping good visual quality. It works by refining video frames in two stages, reducing errors and improving stability in camera motion over time.
What this means in practice
- •For visual effects artists: Create videos with precise camera movements without needing large, annotated datasets or model retraining.
- •For game developers: Generate camera-controlled cutscenes or background videos that follow exact camera paths to improve game immersion.
Authors
Amine Ouasfi, Runjia Li, Junlin Han, Eric Marchand, Philip H. S. Torr, Adnane Boukhayma
Abstract
We present PartiCam, a training-free Particle filtering rooted method for improved Camera controlled video generation. Generating videos that follow a precisely specified camera trajectory remains challenging for large video diffusion models. Training-free approaches are backbone-agnostic and avoid the need to construct large camera-annotated datasets by steering pretrained models toward the desired camera motion at test time. This enables the generation of camera-controlled video data that can subsequently be used to train camera-conditioned video diffusion models. Existing sampling-based guidance approaches often suffer from unstable trajectories: they either explore too broadly and fail to respect the target camera motion or collapse early and lose visual diversity over time. We introduce a global-local refinement framework for diffusion reward guidance, enabling accurate and consistent camera control during video generation. Our method builds on Sequential Monte-Carlo (SMC) guidance, but introduces a local refinement stage based on particle filtered resampling. Experiments show large improvements in camera trajectory adherence, reduced drift, and better visual quality, without requiring model retraining.