SetPlanner automates point-based tool segmentation in surgical images
SetPlanner: A Lightweight Plug-in Point-Set Planner for Frozen SAM
Computer Vision and Pattern Recognition
Summary
Segment Anything Models (SAM) need user input to identify instruments in images, which is hard for tricky tools that are thin, shiny, or partly hidden. The authors developed SetPlanner, a small add-on that plans where to place points automatically without changing SAM’s main setup. This helps SAM guess the tool locations more accurately by generating multiple candidate point sets and picking the best consensus. The system works well on several medical image datasets and improves automatic instrument segmentation without needing ground-truth labels.
What this means in practice
- •For medical imaging teams: Integrate automatic point-based prompts into surgical image segmentation to improve tool detection without manual input.
- •For computer vision engineers: Add a lightweight plug-in planner to existing segmentation models to generate better input points and improve segmentation quality.
Authors
Dawei Yan, Yuezhe Yang, Menglan Ruan, Chunfeng Yang, Yudong Zhang
Abstract
Segment Anything Models provide reusable priors, yet they require user prompts and cannot support fully automatic instrument segmentation. Automatic prompting is difficult for thin, articulated, reflective, and partly occluded tools, where several configurations can be valid. We formulate automatic prompting as lightweight point-set planning and isolate the point source under a frozen pathway. To this end, we present SetPlanner, a 1.52M-parameter plug-in point-set planner for frozen SAM. The plug-in preserves SAM's point-prompt interface and enables reuse across backbones. SetPlanner plans complete unordered K-point sets from geometry-aware targets with a permutation-aware conditional flow. SAM decodes eight candidates; their consensus readout yields a ground-truth-free prediction. Across three endoscopic datasets, SetPlanner wins all six transfer routes over a LoRA-adapted system. Under our frozen-pathway protocol, SetPlanner reaches 0.934 Dice on Kvasir-Instrument and recovers 96% of a 44.4-point localization gap, while candidate disagreement ranks low-Dice cases at AUROC 0.969.