3D scene inpainting works without known camera poses using new neural model
Pose-Free Feed-Forward 3D Inpainting via Learnable Mask Attention and Support Token Refinement
Computer Vision and Pattern Recognition
Summary
Filling in missing parts of 3D scenes usually needs exact camera positions, which can be hard to get in everyday photos. The authors created a method called FreeInpaint that can fill in these gaps without needing camera pose information. It uses a special way to focus on visible areas and gradually guesses what belongs in the missing parts, improving the 3D scene all at once. Their tests show it works well and quickly on various scenes without extra setup.
What this means in practice
- •For 3d graphics developers: Create complete 3D scenes from casual photo sets without needing pre-calibrated camera positions, speeding up scene editing workflows.
- •For augmented reality teams: Improve AR scene reconstructions by automatically filling occluded or missing areas from arbitrary camera views without prior pose calibration.
Authors
Jingyi Pan, Dan Xu, Qiong Luo
Abstract
3D scene inpainting aims to recover missing or occluded regions in edited 3D scenes, while ensuring geometric and textural consistency. Existing approaches, however, typically require accurately calibrated camera poses, which restricts their applicability in casual, in-the-wild scenarios and introduces additional preprocessing overhead. To overcome this limitation, we present FreeInpaint, a novel feed-forward framework that generates complete and 3D-consistent scenes directly from unposed multi-view images with masked regions. At its core, FreeInpaint extends a 3D foundation model to propagate masked regions from a reference view to other unposed views, bridging 3D reconstruction and scene inpainting while preserving the model's native ability to recover camera poses and scene geometry. Our method addresses two key challenges in adapting feed-forward 3D foundation models to masked inputs. First, masked regions can corrupt cross-view correspondence reasoning, degrading pose estimation and geometry recovery. To address this, we introduce a Learnable Mask Attention mechanism that preserves the spatial anchoring of reliable observations while allowing masked regions to progressively absorb useful context in deeper layers. Second, under severe occlusions, a single forward pass often lacks sufficient appearance evidence for high-fidelity completion. Therefore, we propose a Support Token Refinement strategy, which injects diffusion-generated support evidence as confidence-weighted auxiliary tokens to refine under-observed regions while preserving the original spatial anchor. Extensive experiments across diverse datasets demonstrate that FreeInpaint achieves superior inpainting quality, eliminating the reliance on pre-computed camera poses while keeping a fast inference speed. The project page is https://rorisis.github.io/FreeInpaint/.