Colon3r reconstructs 3d colon shape from single video camera
Colon3R: Cross-Domain 3D Reconstruction from Monocular Colonoscopic Video
Computer Vision and Pattern RecognitionRobotics
Summary
Creating 3D images of the colon from single-camera colonoscopy videos is hard because the inside of the colon has smooth, shiny surfaces and moves a lot. The authors developed Colon3R, a method that learns from fake and lab data and then improves itself by using unlabeled real colonoscopy videos. Colon3R selects reliable information carefully to better guess the 3D shape and camera position, even without detailed real data labels. Their tests show Colon3R makes more accurate and complete 3D reconstructions than previous methods on real colonoscopy videos.
What this means in practice
- •For surgical robotic teams: Improve 3D mapping of colon shape during robotic colonoscopy to enhance navigation and safety without needing extensive real data labels.
- •For medical imaging developers: Develop software that generates more complete and accurate 3D models from colonoscopy videos despite challenging visual conditions.
Authors
Zhihao Xing, Yingyu Wang, Liang Zhao, Shoudong Huang
Abstract
Monocular colonoscopic 3D reconstruction is important for surgical robotic colonoscopy, but remains challenging due to weak texture, specular reflections, limited view overlap, and non-rigid tissue motion. Conventional multi-view 3D reconstruction methods rely on stable correspondences and approximate rigidity, which are often violated in colonoscopy. Existing endoscopic methods often rely on domain-specific supervision, whereas there are not enough in-vivo labeled data available to adapt geometry foundation models to clinical colonoscopy. We present Colon3R, a cross-domain semi-supervised framework built on pretrained VGGT that transfers coupled camera, depth, and pointmap geometry from labeled phantom and simulated data to unlabeled in-vivo colonoscopy without requiring target-domain geometric annotations. Unlike source-only fine-tuning, which learns only from phantom and simulated data, Colon3R directly exploits unlabeled in-vivo video through teacher-derived cross-view supervision. Our proposed hierarchical quasi-rigid reliability selects reliable supervision at the sequence, directed-pair, and pixel levels, while source-preserving adaptation retains the learned coupled geometry during target-domain adaptation. Extensive experiments demonstrate that our method achieves superior overall performance over state-of-the-art approaches in depth, pointmap, and camera pose estimation. Qualitative comparisons on real in-vivo colonoscopy further show substantially more complete and geometrically consistent reconstructions than competing methods under clinical domain shift. The code will be public available after the paper is accepted.