Surgical instrument segmentation improves using reasoning and queries
SIRA: Reasoning-Aware Surgical Instrument Segmentation via Query-Anchored Alignment
Computer Vision and Pattern Recognition
Summary
Identifying surgical instruments in images is important for robotic surgery and workflow analysis but often misses the procedure context. The authors propose a new way to segment instruments that understands specific questions about the surgery images, improving how well instruments are recognized. They created a large dataset pairing images with detailed questions and answers to help train and test this approach. Their new system, called SIRA, uses both the surgical context and visual clues to better predict where instruments are in images.
What this means in practice
- •For robotic surgery teams: Improve identification and tracking of surgical tools during procedures using query-driven segmentation that adapts to specific surgical contexts.
- •For medical imaging software developers: Enhance surgical workflow analysis tools by integrating spatial and semantic reasoning for more accurate instrument segmentation.
Authors
Zhibo Zhang, Qijie Wang, Zengqiang Yan
Abstract
Surgical instrument segmentation (SIS) plays a critical role in robotic assistance and surgical workflow analysis. However, most existing SIS methods formulate segmentation as a category-driven localization problem, limiting their ability to capture procedural context and task-dependent semantics in surgical workflows. We introduce Reasoning-Aware Surgical Instrument Segmentation (RA-SIS), a task formulation that frames segmentation as query-conditioned inference under surgical context. To benchmark this setting, we construct SurgRS, a surgical reasoning segmentation dataset consisting of 41,000 image-text pairs, which aligns instance-level masks with structured query-answer supervision to enable semantic grounding at the pixel level. Based on SurgRS, we propose Surgical Instrument Reasoning and Segmentation Assistant (SIRA), a multimodal framework that disentangles target-level and query-level semantics and integrates them with visual features through query-anchored dual alignment. By aligning query semantics with spatial features and segmentation prompts, SIRA enhances semantic-visual consistency in mask prediction. Extensive experiments on SurgRS demonstrate improvements over existing reasoning-aware baselines. Code is available at https://github.com/linxir226/SIRA.