3D method turns moving-camera videos into seamless 360-degree views

A 3D-Driven Framework for Moving-Camera Perspective-to-360° Video Generation

Computer Vision and Pattern Recognition

Summary

Turning a video taken with a moving camera into a smooth 360-degree video is tricky because you have to fill in parts you didn't see, keep things consistent, and maintain what was already recorded. The authors of this work created a system that handles these challenges step-by-step using a 3D model to represent the scene. They use a separate tool to fill in missing areas and then integrate everything into the 3D space so it all stays in the right place when the viewpoint changes. Their approach works well without needing extra training on 360-degree videos and produces good quality and consistent results.

What this means in practice

  • For virtual reality developers: Convert standard moving-camera footage into consistent 360-degree videos for immersive VR experiences without retraining on panoramic data.
  • For film post-production teams: Enhance footage by generating missing content and maintaining spatial consistency when transitioning shots to panoramic or VR formats.

Authors

Michael Wakeham, Mrinmoy Kumar Das, Wenqi Wang, Ayush Tewari, Boqing Gong, Deepti Ghadiyaram

Abstract

Turning a moving-camera perspective video into a 360° video requires more than just generating content outside the input frame: observations revealed over time must be preserved, unseen regions completed, and the resulting scene kept consistent as the camera viewpoint changes. Existing methods assign these different requirements to a single conditional video model, requiring them to be jointly learned from panoramic video data. We instead present a modular, 3D-driven framework that distributes these three responsibilities across specialized components, all connected through a shared scene representation. Input observations are reconstructed explicitly in 3D, while a pretrained panoramic image model completes unobserved content from a selected panoramic render. This generated content is lifted into the shared 3D scene, where it remains spatially anchored across camera viewpoints. Without requiring any additional panoramic-video training, our framework achieves state-of-the-art input-view fidelity, perceptual quality, and geometric consistency on a static-scene subset of the Argus evaluation set. These results support our decomposed, 3D-driven formulation as a promising alternative to direct video generation in this setting.