Semantic rgb depth boosts surgical skill classification from microscope videos
Semantic RGB--Depth Based Surgical Skill Assessment in Microscopic Stereo Videos
Computer Vision and Pattern Recognition
Summary
Measuring how well surgeons perform delicate procedures is important for training and safety. The authors found that adding 3D depth information from stereo microscope videos helps computers better recognize surgeons’ skill levels than using just regular 2D video. They combined sparse 3D data with detailed 2D images to create richer video representations. Their system could distinguish skill levels more accurately and also highlight important parts of the videos for review.
What this means in practice
- •For hospital training teams: Use the combined RGB-depth video analysis to objectively assess and differentiate surgeon skill levels during microsurgery training sessions.
- •For surgical equipment developers: Integrate enhanced depth-aware video processing into surgical microscopes to support real-time skill evaluation and feedback for surgeons.$Commercial implications: Enables sale of surgical microscopes with built-in skill assessment features, appealing to training hospitals and medical centers.
Tested on one dataset.
Authors
Jecia Z. Y. Mao, Sue M. Cho, Francis X. Creighton, Deepa Galaiya, Russell H. Taylor, Manish Sahu
Abstract
Objective assessment of microsurgical technical skill is essential for competency-based training and quality assurance, yet existing video-based approaches predominantly rely on RGB images and therefore overlook the 3D spatial relationships that characterize instrument-anatomy interactions. Although stereo operating microscopes provide complementary depth information, conventional stereo matching algorithms can produce sparse and unreliable depth estimates under high-magnification imaging conditions, limiting their use for automated skill assessment. This work presents a semantic RGB-Depth framework for surgical skill assessment from microscopic stereo videos. A regression-based depth fusion method combines sparse metric stereo depth with dense monocular depth estimates to generate a dense geometric representation of the surgical scene. This representation is integrated with semantically decomposed RGB streams corresponding to individual surgical instruments and surrounding anatomy. A hierarchical attention architecture jointly encodes these streams to capture discriminative patterns of instrument use and instrument-anatomy interaction across surgeons at different training levels. The framework was evaluated on 33 ex vivo transoral microlaryngeal procedures performed by six surgeons, comprising attending surgeons and surgical residents, using leave-one-surgeon-out cross-validation. The proposed semantic RGB-Depth model achieved an F1 score of 0.938 for skill-level classification, compared with 0.696 for semantic RGB and 0.929 for semantic depth. These results suggest that geometric information can improve automated surgical skill assessment from microscopic stereo videos. The learned spatial, temporal, and semantic attention patterns also support qualitative examination of the scene regions, video segments, and semantic streams emphasized by the model.