Multimodal robotic system improves real-time surgical navigation accuracy
Multimodal Image-based Robotic Assistance and Guidance Environment (MIRAGE): A Novel Unified Framework for Surgical Navigation and Visualization
Robotics
Summary
Surgical robots need precise and real-time guidance to help surgeons operate safely. The authors developed MIRAGE, a system that combines multiple imaging techniques like 3D stereo vision, ultrasound, and augmented reality in one framework to assist surgeons during operations. They tested it on several versions of the da Vinci surgical robot, showing that it can provide very accurate visual guidance with errors of less than a millimeter. This technology helps surgeons see important information without interrupting their workflow, potentially making surgeries safer and more efficient.
What this means in practice
- •For robotic surgery teams: Enable precise and integrated visual guidance combining 3D video, ultrasound, and AR overlays during robot-assisted surgeries using the da Vinci system.
- •For medical device developers: Build extensible multimodal surgical assistance platforms that unify calibrated image and tracking data streams for real-time intraoperative use.$Commercial implications: This framework supports advanced surgical visualization products that improve operator interface and accuracy in robotic surgery consoles.
Authors
Ehsan Nasiri, Yi Zheng, Michael A. Kokko, Xiaoyao Fan, William R. Warner, Keith D. Paulsen, Ryan J. Halter
Abstract
This paper presents the computational architecture and quantitative preclinical evaluation of MIRAGE, a multimodal image-guided robotic surgery framework for real-time visualization and navigation. It integrates stereovision scene reconstruction, augmented-reality (AR) video overlay, quantified heads-up display (HUD) navigation, and intraoperative ultrasound within the da Vinci console's TilePro display (Intuitive Surgical). Additional modules include optical/electromagnetic tracker registration, stereo calibration, fiducial rigid registration with RMSE feedback, and 3D point-cloud reconstruction. Video I/O and rendering run in a C++ backend, while a Python GUI coordinates calibration, registration, reconstruction, visualization, and user interaction through an in-process zero-copy binding layer. The framework formalizes spatial transforms, module states, and synchronization of heterogeneous video and tracking streams. Preoperative calibration uses synchronized checkerboard image pairs and optimization, with estimated intrinsics, extrinsics, and rectification reused intraoperatively. MIRAGE was validated in phantom experiments across da Vinci S, Si, Xi, and SP platforms. Across clinically relevant working distances, mean 3D reprojection error was 0.6mm, with best 0.1mm RMSE at 100mm on Xi. Results support MIRAGE as a preclinical technical-validation platform for accurate, responsive, and extensible multimodal surgical guidance.