Endoscopic reconstruction improved with temporal spectral appearance modeling
Endo-TSR: Temporal Spectral Modeling of Appearance and Motion for Endoscopic Reconstruction
Computer Vision and Pattern RecognitionRobotics
Summary
Reconstructing 3D images inside the body during endoscopy is tricky because tissues move and look different over time. The authors created Endo-TSR, a method that captures both how tissues move and how their appearance changes using specialized mathematical models. This helps create clearer and more detailed images, which they tested on medical datasets and showed better quality than existing methods. Their work also showed that modeling appearance changes over time was key to improving the results.
What this means in practice
- •For medical imaging developers: Add temporal appearance modeling to endoscopic 3D reconstruction tools to enhance image clarity and surface detail recovery during tissue motion.
- •For computer vision engineers: Integrate bounded Fourier color residuals with deformable Gaussian splatting for improved modeling of dynamic scenes with changing appearance.
Authors
Taoyu Wu, Yiyi Miao, Qi Shao, Zhuoxiao Li, Zhe Tang, Limin Yu, Baoru Huang
Abstract
Endoscopic scene reconstruction requires modeling tissue motion and temporal appearance while recovering fine surface detail. Deformable Gaussian models provide explicit trajectories, but their fixed colour coefficients lack a dedicated temporal representation for photometric changes. We propose Endo-TSR, which augments deformable Gaussian splatting with bounded Fourier colour residuals and independent translation residuals on shared temporal frequencies. The colour residuals capture local appearance changes, while a Matérn spectral prior regularises motion corrections. Multi-scale Laplacian supervision guides tissue-detail recovery during joint image fitting. Extensive experiments on the EndoNeRF and StereoMIS datasets demonstrate state-of-the-art rendering quality, with the highest PSNR across all evaluated sequences. Ablation studies show that temporal appearance yields the largest PSNR gain among the tested component additions, while appearance and detail supervision jointly improve rendering with fixed Gaussian counts.