Surgical robot instrument tracking improves with self-training fusion
Robust Surgical Robotic Instrument Tracking via Sequential Multi-Cue Fusion and Sim-to-Real Self-Training
RoboticsComputer Vision and Pattern Recognition
Summary
Tracking surgical tools during robot-assisted surgery is hard because the tools look different and the surgery scene is complex. The authors developed a system that learns from computer-generated images and then improves itself by watching real surgery videos without needing costly manual labels. It uses a smart filter to combine multiple clues about the tool’s position and then fine-tunes a keypoint detector to track instruments more accurately. Their experiments show this approach works better and faster than previous methods.
What this means in practice
- •For surgical robotics developers: Enhance real-time tracking of robotic surgery tools using self-trained models on unlabeled video data to improve accuracy without needing extensive manual annotations.
- •For medical device integrators: Incorporate efficient instrument tracking software that adapts from synthetic training to real operating room videos for better robot-assisted surgery outcomes.
Authors
Hanyang Hu, Zekai Liang, Florian Richter, Michael C. Yip
Abstract
Efficient and robust tracking of surgical robotic instruments is important for robot-assisted minimally invasive surgery, yet remains challenging due to the complexity of surgical scenes and the unconventional geometry of surgical instruments. Keypoint-based approaches are efficient, but their performance depends on reliable feature detection. Improving these detectors with real-world supervision is difficult because accurate real-world annotations are costly to obtain at scale. To address this limitation, we introduce a tracker-guided self-training framework that adapts a model pretrained on synthetic images to unlabeled real-world videos. Given measured robot joint states, an uncertainty-aware EKF recursively corrects the instrument pose and the observable joint angles by comparing projected model features with detected keypoints, shaft boundaries, and mask-derived cues. An RTS smoother subsequently refines the resulting trajectory, which is projected into pseudo-labels for fine-tuning the feature detector without laborious pose annotations. Experiments on real-world videos demonstrate consistent improvements from self-training across all evaluated keypoint metrics, and the resulting model outperforms prior approaches in both accuracy and runtime. The code and data will be released upon publication.