Surgical tool pose estimation improves with projection and consistency optimization
PICO: Projection-Informed Consistency Optimisation for 6DoF Surgical Tool Pose Estimation
Computer Vision and Pattern Recognition
Summary
Estimating the exact position and angle of surgical tools during operations is important for safe robot-assisted surgery. Existing methods either rely on arm mechanics that can accumulate errors or vision systems that need markers and can be slow. The authors propose a new AI model called PICO that learns to predict the tool’s shape and position together, using geometric rules to improve accuracy. Tests show PICO works well even when parts of the tools are hidden.
What this means in practice
- •For robotic surgery teams: Enable real-time, marker-free estimation of surgical tool positions even during occlusions to enhance robot control and safety.
- •For medical device developers: Incorporate a trainable model improving tool pose robustness into surgical robots to reduce reliance on external trackers and fixed markers.$Commercial implications: This paper enables the development of integrated vision systems for surgical robots that competitors could sell as enhanced navigation components.
Authors
Lucy Fothergill, Pietro Valdastri, Dominic Jones, Duygu Sarikaya
Abstract
Purpose: Accurate 6 DoF pose estimation of surgical tools is critical for automa- tion, robotic proprioception, and safe interaction with the tissue operated on. Kinematics-based approaches suffer from accumulated errors due to the cable- driven nature of robotic arms, while vision-based methods often rely on external markers or trackers. Although more recent vision-based advances have been pro- posed, these two-stage pose estimation methods often lack real-time robustness due to accumulated errors and computational overhead. Methods: We propose a novel end-to-end trainable model, PICO. Our model employs a multi-task learning architecture to predict segmentation and depth maps, alongside regression of translation and rotation parameters. We define two proxy tasks that enforce geometric consistency in both 2D and 3D spaces, improving accuracy and robustness. For this, we propose a projection loss, and a point-to-point loss. Results: We evaluate our method on the SurgRIPE dataset, benchmarking its performance against state-of-the-art approaches using standard 6DoF pose esti- mation metrics. Our results demonstrate consistently strong performance across all four subsets, specifically in rotation, ranking second even under occlusion. It also demonstrates comparable translational performance, remaining competitive, especially in occluded cases. Conclusion: PICO demonstrates the effectiveness of multi-task learning and geometry-aware proxy tasks for robust and reliable surgical tool pose estimation, especially in occluded scenarios, highlighting potential for future applications.