Surgical phase recognition improves accuracy and stabilizes workflow predictions
Shifted Window Temporal Transformer for YOLO-Guided Phase Recognition in Laparoscopic Cholecystectomy Surgery
Computer Vision and Pattern Recognition
Summary
During surgeries, it’s important for computer systems to know exactly what step the surgeons are on to help with decisions and teamwork. Previous computer methods sometimes got confused when the surgery moved from one step to another. The authors created a new method combining advanced image and time-based analysis to watch surgeries and detect tools at the same time. Their system gave very accurate and stable results, better than previous methods, on a standard dataset of gallbladder removal surgeries.
What this means in practice
- •For hospital it teams: Integrate the method into operating rooms to provide real-time updates on surgical phases and instrument locations during laparoscopic gallbladder removal.
- •For robotic surgery developers: Use the recognition framework to improve workflow understanding and instrument tracking for automated assistance in minimally invasive surgery.
Authors
Sheekar Banerjee, Md. Shahabuddin, Md. Mahbub Hasan Akash, Nabilah, Mohammad Arafat Hussain
Abstract
Intelligent operating rooms require accurate awareness of surgical workflow for better intra-operative decisions, teamwork, and surgical documentation. Therefore, accurately recognizing surgical phase during surgery is crucial. However, previous artificial intelligence-based methods provide fluctuating predictions for the transitions, restricting their clinical use. To address this challenge, we propose a clinically meaningful framework that integrates a Swin Transformer and Temporal Transformer Encoder (TTE) for effective surgical phase recognition and a parallel YOLOv11s network for real-time surgical instrument localization. To avoid phase flickering and to produce stable workflow predictions, a causal temporal smoothing is applied. The proposed framework is tested on the Cholec80 dataset for the simultaneous recognition of the surgical phases and instruments. The overall phase recognition accuracy of this model was 92%, and the macro average F1-score was 0.86. This performance surpasses CNN based methods such as EndoNet (81.7%) and SV-RCNet (85.3%), as well as temporal convolutional and transformer-based approaches including TeCNO (88.6%), Trans-SVNet (90.9%), and LoViT (91.5%). Besides, the module of instrument detection achieved mAP@0.5 = 0.863. The reliability of the results is confirmed, obtaining a high recognition rate of the workflows, and a high accuracy of the localization of the instruments. The proposed framework can serve as a foundation for intelligent surgical assistance, enriching situation awareness, and potential robotic surgery applications. Code will be made publicly available at: https://github.com/ac005sheekar/LapChol-Surgery-Swin-Cholec80