Temporal memory transformer improves surgical instrument segmentation
TEDi: Temporal Memory-Enhanced and Denoising Transformer for Surgical Instrument Segmentation
Computer Vision and Pattern Recognition
Summary
Recognizing surgical tools in video frames is important but can be unreliable when looking at each frame alone. The authors created TEDi, a method that remembers information from earlier frames and uses this memory to improve current predictions. TEDi also checks for consistency across frames to reduce errors and make the tool recognition smoother. Tests on medical video datasets show TEDi works better than previous methods.
What this means in practice
- •For medical imaging developers: Improve surgical video analysis by using temporal memory to segment instruments more accurately across frames.
- •For robotic surgery engineers: Enhance robot-assisted surgery systems' understanding of instrument position and identity with temporally consistent segmentation.
Authors
Jiahong Yuan, Weiming Mi, Tao Zhang, Haoyin Zhou
Abstract
Query-based segmentation methods have shown promising potential for surgical instrument segmentation and recognition, which is essential for scene understanding and downstream tasks in computer assisted surgery. However, most existing approaches predominantly rely on per-frame predictions and overlook cross-frame temporal priors as well as temporal-consistency constraints. This limitation often leads to unstable query representations and suboptimal category recognition. In this paper, we propose TEDi, a Temporal memory-Enhanced and Denoising transformer for surgical instrument segmentation that addresses these is sues through Memory Search Enhancement and Temporal Consistency Denoising. The former introduces a query-level memory bank and a memory search enhancement encoder to retrieve discriminative representations from historical frames, enriching current-frame features. The latter constructs a temporally consistent reference as a cross-frame semantic anchor to suppress temporally unstable predictions and promote semantic coherence across frames. Extensive experiments on two benchmark datasets, EndoVis 2017 and EndoVis 2018, demonstrate that TEDi consistently outperforms state-of-the-art methods, highlighting its potential to further advance computer-assisted surgery. Our code is available at github.com/argon-xixi/TEDi.