Foundation model assesses robotic surgery skills and recognizes gestures
SurGPT: A Multi-Task Foundation Model for Robotic Surgical Skills Assessment and Gesture Recognition
Robotics
Summary
Surgical training often depends on experts watching videos to judge how well someone performs robotic surgery, which can be slow and subjective. The authors developed a computer model that watches surgical videos and can tell what task is happening, how skilled the surgeon is, and describe the specific hand movements in natural language. Their model uses video analysis and a special attention-based design to understand the sequence of actions. It performed well on a surgical simulator dataset and could help make surgical education faster and more objective.
What this means in practice
- •For surgical trainers: Automatically evaluate surgical trainee performance and provide detailed gesture descriptions from robotic surgery videos to streamline training.
- •For surgical simulation developers: Integrate AI-based skill and gesture assessment to enhance feedback features in surgical simulators using recorded procedure videos.
Authors
Parham M. Kebria, Saeid Nahavandi, Mehrdad Saif, Fernando Bello
Abstract
The assessment of surgical skills in robotic surgery training has traditionally relied on subjective evaluations by human experts, creating bottlenecks in surgical education and limiting scalability. We present a novel foundation model that automatically analyzes robotic surgical videos to provide comprehensive skill assessment and gesture understanding. Combining a visual encoder with a transformer architecture, the model performs three key tasks simultaneously: 1) surgical task classification (knot tying, needle passing, suturing), 2) skill level assessment on a standardized scale, and 3) natural language generation of surgical gesture descriptions. The model processes video sequences from the da Vinci surgical robot and generates human-readable text descriptions of surgical actions, similar to how language models generate text from prompts, using a multi-task learning framework with adaptive uncertainty weighting to optimize performance across all objectives. Our transformer-based architecture incorporates positional encoding for temporal understanding and attention mechanisms for spatiotemporal feature integration. Evaluated on the da Vinci Skills Simulator dataset, the proposed framework demonstrates state-of-the-art performance in surgical task recognition and provides interpretable skill assessments that correlate with expert evaluations, with the model's ability to generate natural language descriptions of surgical gestures opening new possibilities for automated surgical education, real-time feedback systems, and objective skill assessment in surgical training and certification programs.