Interactive 3d reconstruction improves with sparse user clicks
Click3R: Interactive Stereo 3D Reconstruction with Sparse Correspondence Clicks
Computer Vision and Pattern Recognition
Summary
3D reconstruction models usually make one guess about the shape of objects from images and cannot fix themselves if wrong. This paper presents Click3R, a method where a person can click on a few matching points between two images to help the model improve its 3D guess. These clicks add helpful geometric clues that fix tricky cases like repeated patterns or when parts of objects look very similar. The authors show that even one click can significantly reduce errors in 3D reconstructions.
What this means in practice
- •For 3d scanning professionals: Correct 3D scans interactively by adding a few user clicks to fix common reconstruction errors.
- •For augmented reality developers: Improve 3D scene models in AR systems on the fly by letting users quickly resolve ambiguities with minimal input.
Authors
Hao Gu, Jin Yao, Zezhou Cheng
Abstract
Despite impressive progress, existing feed-forward large 3D reconstruction models produce a single, irrevocable prediction. When these models fail, there is no mechanism to correct their mistakes. To address this limitation, we introduce Click3R, a framework for interactive geometric correction of feed-forward models using sparse human correspondence clicks. Given an image pair and a small set of cross-view point correspondences provided by a user, Click3R injects geometric constraints into the 3D reconstruction networks through a lightweight point correspondence adapter, enabling the model to resolve ambiguities in challenging scenarios such as repetitive textures, symmetric structures, large viewpoint changes, or textureless regions. To evaluate interactive reconstruction, we curate a challenging benchmark specifically designed to expose failure modes of existing feed-forward 3D reconstruction methods. Experiments show that Click3R reduces reconstruction error with as few as a single correspondence click, improving performance on both standard and our curated benchmarks. Our results highlight that sparse human interaction provides an effective and practical mechanism for correcting geometric reconstruction errors.