Video reshooting uses 3D scene paths to guide video generation
Holo4D: Holistic 4D Reconstruction as Geometric Control for Video Diffusion
Computer Vision and Pattern Recognition
Summary
Creating videos from new angles is hard because parts of the scene that were hidden before need to be filled in realistically. The authors combine 3D scene reconstructions, which capture how objects move and look from the original view, with a video generation model that can imagine missing parts. Their method uses the 3D paths as a guide to reshoot videos from different camera positions, blending real observed content with smart guessing for unseen regions. This approach lets video generation better follow real scene geometry and motion while still creating new views.
What this means in practice
- •For video production teams: Create new video shots from different camera angles using a guiding 3D scaffold, improving realism in visual effects and reshoots.$Commercial implications: Enables production companies to offer advanced video reshooting services that combine real scene data with generative video synthesis.
- •For augmented reality developers: Generate consistent video content from changing perspectives by integrating reconstructed 3D motion paths to enrich AR experiences.
Authors
Yushi Lan, Zeren Jiang, Koichi Namekata, Xingang Pan, Chuanxia Zheng, Andrea Vedaldi
Abstract
Synthesizing a monocular video from a new camera trajectory requires preserving the observed scene geometry and motion while completing regions that become newly visible. Feed-forward 4D reconstructors recover the structure of the observed scene, whereas video diffusion models provide a generative prior for completing unseen content. We introduce Holo4D, a video reshooting framework that uses reconstructed 3D trajectories as a holistic source-view 4D scaffold. We project these trajectories into the requested views and encode their geometry and source appearance to guide a pretrained video diffusion model. Together with the source video and target cameras, the scaffold grounds generation in reconstructed observations while leaving unsupported regions to the generative prior. Experiments demonstrate the value of trajectory-based conditioning for camera control and novel-view synthesis. The framework accommodates different 4D reconstructors without retraining the video generator, allowing it to benefit from future advances in source reconstruction. More details can be found in our project page.