Syncvid generates synchronized videos from one reference across views
SyncVid: World-Equivariant Diffusion for Synchronized Video Generation
Computer Vision and Pattern Recognition
Summary
Generating videos of a moving scene from one video is tough because new views must match the original's moving parts closely. The authors solve this by creating shared noise patterns in a 3D space instead of random noise for each view separately, keeping videos in sync. Their method turns a single video into a 4D format and uses the geometry of the scene to guide video generation. This improves how well different views of the same scene match each other over time without changing the video model's design.
What this means in practice
- •For visual effects teams: Create multiple synchronized video angles from one reference shoot to maintain dynamic scene consistency.
- •For virtual reality developers: Generate consistent videos from different viewpoints for immersive environments using a single source video input.
- •For game animation studios: Produce synchronized dynamic scene videos across views for richer in-game cinematics without extra capture sessions.$Commercial implications: Enables selling better multi-view synchronized video generation tools based on a single capture, improving animation pipelines.
Authors
Farhad G. Zanjani, Animesh Karnewar, Fatih Porikli, Amirhossein Habibian
Abstract
Generating multiple videos of a dynamic scene from a single reference video requires synthesizing both observed and unobserved content consistently across viewpoints. Existing camera-control methods for video diffusion typically rely on architectural mechanisms, including camera embeddings, cross-view attention, and geometric conditioning. Although effective for static scene content, these approaches often struggle with the fine-grained, dynamic geometry that must remain coherent across views and over time. We synchronize the diffusion process itself by sampling the Gaussian noise prior in a canonical 3D space derived from the source video and rasterizing it into target cameras, rather than sampling independently in each 2D image plane. Our method, SyncVid, lifts a monocular video into a canonical 4D representation, associates noise with scene geometry and unseen space, and rasterizes the resulting structured noise into arbitrary views. Because rendering can introduce local correlations, we further propose a cross-view-consistent whitening procedure that decorrelates neighboring pixels while preserving correspondence across views. The structured noise serves as the diffusion prior for a LoRA-adapted pretrained video diffusion model, requiring no architectural modifications. By enforcing spatiotemporal and cross-view coherence at the source of stochasticity, SyncVid generates synchronized videos and improves both perceptual quality and cross-view consistency in our experiments. Project page: https://qualcomm-ai-research.github.io/SyncVid-Webpage.