Orca improves 3D scene views by fixing missing image parts
ORCA: Occlusion-Aware Refinement and Completion for Novel View Synthesis
Computer Vision and Pattern Recognition
Summary
Making 3D views from just one picture is tricky because when you move around, you see parts hidden before that aren’t in the original photo. The authors showed that many small gaps in these views can be fixed by reusing information already in the 3D scene, rather than guessing new details. Their method, called ORCA, decides if missing parts are small or large, fixing small holes with existing data and only inventing new content for big gaps. This reduces errors and keeps the scenes looking like the real photo.
What this means in practice
- •For computer graphics developers: Create more accurate navigable 3D scenes from a single image by repairing small hidden regions using existing 3D data before resorting to image generation.
- •For virtual reality content creators: Generate better visual quality for virtual explorations by limiting artificial image generation and preserving true scene details in novel viewpoints.
Authors
Weronika Jakubowska, Maciej Zięba, Przemysław Spurek
Abstract
Novel-view synthesis from a single image is a fundamentally ambiguous problem. As the camera moves away from the input viewpoint, previously hidden regions become visible, exposing missing geometry and holes in the reconstructed scene. Existing methods often rely on generative models to complete such regions. However, many of these artifacts are small gaps near depth boundaries and do not require generating new scene content. In order to eliminate expensive process of generating image we introduce ORCA, an occlusion-aware method for reconstructing and completing explorable 3D scenes from a single image. ORCA first introduces 3D structure into a Gaussian-anchor representation using monocular depth while preserving the original camera-ray correspondence. During scene exploration, missing regions are handled based on their size and structure. Small disocclusions are repaired using RGB-D information already available in the reconstruction, while generative inpainting is reserved for larger regions that cannot be reliably recovered from the scene. New Gaussian anchors are added and optimized locally without modifying the existing representation. By reducing unnecessary reliance on generative inpainting, ORCA limits generation-induced hallucinations and better preserves the content and structure of the original scene. On DIV2K, ORCA improves novel-view quality over VistaDream across all reported metrics, increasing MUSIQ from 61.60 to 68.71 and CLIP-IQA from 0.474 to 0.574. These results show that many novel-view artifacts can be repaired effectively by reusing information already present in the reconstructed scene.