Multimodal AI improves surgical tool bounding box labeling at scale
From Presence Labels to Bounding Boxes: Can Multimodal Large Language Models Scale Surgical Instrument Localization?
Computer Vision and Pattern Recognition
Summary
Identifying exactly where surgical tools are in video frames is important for understanding operations, but manually drawing boxes around them takes a lot of expert time. The authors show how large AI language models that use both images and text can turn simple presence labels into accurate tool boxes automatically. Their method performs well compared to human annotators and produces a large new dataset of tool locations in surgical videos. This could help improve future surgical AI systems without needing exhaustive manual labeling.
What this means in practice
- •For medical imaging engineers: Automatically annotate surgical videos with precise tool bounding boxes to train and evaluate computer vision models for surgical scene understanding.
- •For robotic surgery developers: Use enriched surgical instrument localization data to improve tracking and interaction analysis for robot-assisted surgeries.
Tested on one dataset.
Authors
Mohamed Hamdy, Muraam Abdel-Ghani, Fatmaelzahraa Ahmed, Sifna Nasar, Mariam Ahmed, Khalid Al-Jalham, Abdulaziz Al-Ali, Shidin Balakrishnan
Abstract
Spatial localization of surgical instruments through bounding-box annotations is a foundational component of surgical scene understanding, supporting detection, tracking, interaction analysis, and grounded vision-language applications. Yet existing annotations often cover selected frames, video subsets, or dataset variants rather than complete procedures. Manual extension is time-consuming and expert-dependent. Additionally, weakly supervised and detector-based pseudo-labels require task-specific training and may be imprecise. In this study, we design a scalable MLLM-based annotation framework that converts surgical frames and existing instrument-presence labels into accurate bounding boxes, and systematically analyze its reliability, cost, and design sensitivity. We instantiate the framework using six MLLMs and a multi-call configuration on laparoscopic cholecystectomy in Cholec80. Using a carefully designed stratification scheme, two annotators independently label 1,000 frames, and an expert surgeon reviews one annotation set to establish the reference. Results show that instrument-set F1 ranges from 88.2 to 92.7,whereas mIoU ranges from 0.180 to 0.735, showing that comparable recognition can conceal large localization differences. A structured multi-stage annotation pipeline achieves 0.784 mIoU, compared with 0.735 for the best single-call baseline and 0.814 human-human agreement. Using the best accuracy pipeline, we create and publicly release a spatial extension of Cholec80 comprising 285,977 bounding boxes across all 184,498 annotated frames from the 80 videos. The annotations and human references are made available on GitHub