Stitch-Inferencer: Enhance Endoscopic Video Segmentation and Tracking via Panoramic Reconstruction
Computer Vision and Pattern Recognition
Summary
The authors address challenges in understanding surgical videos due to limited camera views and tools blocking the sight. They created Stitch-Inferencer, which builds a big picture by stitching together clear parts from multiple video frames, helping to see areas hidden in any single frame. This approach works with existing models without retraining and runs fast enough for real-time use during surgeries. Tests showed it improves tasks like identifying anatomy and tracking instruments while keeping the system efficient.
Authors
Shunsuke Kikuchi, Atsushi Kouno, Hiroki Matsuzaki
Abstract
Surgical video understanding is fundamental to navigation systems. Endoscopic perception is often hindered by a limited field-of-view and frequent instrument occlusions, making spatio-temporal context essential for robust inference. These challenges have motivated video models that aggregate information across frames. However, existing video models typically store past observations implicitly in learned feature representations, often requiring task-specific video training, substantial annotated data, and increased computational cost. We propose Stitch-Inferencer, a real-time, model-agnostic inference framework that replaces implicit feature memory with an explicit image-space panoramic canvas. By stitching valid observations across frames, Stitch-Inferencer preserves previously observed pixels in an online, instrument-free view, expanding the effective field-of-view and providing direct access to regions that are temporarily occluded or absent from the current frame. Downstream segmentation or tracking models are applied to a compact region of interest on the panorama, and their predictions are reprojected to the current frame, enabling existing models to exploit long-range context without retraining. Experiments on anatomy segmentation and point/box tracking demonstrate consistent improvements across diverse baselines while preserving real-time throughput. The stitching module alone runs at over 60 FPS, providing a practical inference-time solution to enhance endoscopic perception in computationally constrained intraoperative environments. Source code will be made publicly available.