Event-Level Detection of Surgical Instrument Handovers in Videos with Interpretable Vision Models
2026-04-08 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors developed a computer model to automatically detect when surgeons pass instruments to each other during operations by analyzing surgical videos. Their method combines a type of image processor called a Vision Transformer with a memory network that understands sequences, allowing it to spot when and how instruments are handed over. They tested the model on kidney transplant surgeries and found it accurately detected these handovers and the direction of transfer. Additionally, they used a visualization technique to show which parts of the images influenced the model's decisions, helping to understand how it works.
Vision Transformer (ViT)Long Short-Term Memory (LSTM)surgical instrument handoverevent detectionmulti-task learningtemporal aggregationpeak detectionLayer-CAMintraoperative video analysis
Authors
Katerina Katsarou, George Zountsas, Karam Tomotaki-Dawoud, Alexander Ehrenhoefer, Paul Chojecki, David Przewozny, Igor Maximilian Sauer, Amira Mouakher, Sebastian Bosse
Abstract
Reliable monitoring of surgical instrument exchanges is essential for maintaining procedural efficiency and patient safety in the operating room. Automatic detection of instrument handovers in intraoperative video remains challenging due to frequent occlusions, background clutter, and the temporally evolving nature of interaction events. We propose a spatiotemporal vision framework for event-level detection and direction classification of surgical instrument handovers in surgical videos. The model combines a Vision Transformer (ViT) backbone for spatial feature extraction with a unidirectional Long Short-Term Memory (LSTM) network for temporal aggregation. A unified multi-task formulation jointly predicts handover occurrence and interaction direction, enabling consistent modeling of transfer dynamics while avoiding error propagation typical of cascaded pipelines. Predicted confidence scores form a temporal signal over the video, from which discrete handover events are identified via peak detection. Experiments on a dataset of kidney transplant procedures demonstrate strong performance, achieving an F1-score of 0.84 for handover detection and a mean F1-score of 0.72 for direction classification, outperforming both a single-task variant and a VideoMamba-based baseline for direction prediction while maintaining comparable detection performance. To improve interpretability, we employ Layer-CAM attribution to visualize spatial regions driving model decisions, highlighting hand-instrument interaction cues.