Digital twin vr system improves surgical robot teleoperation accuracy
Digital Twin-Driven VR Teleoperation with Multi-View Spatial Perception for Surgical Robots
Robotics
Summary
Surgical robots help doctors operate inside the body through tiny cuts, but current control setups only show the surgery from one view, making it hard to judge depth. The authors created a virtual reality system that uses a digital copy of the surgery area, letting surgeons see multiple angles and move naturally. This system also uses VR hand controllers for more precise and stable control compared to previous methods. Their tests showed the new system lets surgeons move instruments more smoothly and judge depth better than existing mixed reality systems and traditional consoles.
What this means in practice
- •For robotic surgery teams: Use a VR digital twin platform to improve precision and depth perception during minimally-invasive robot-assisted surgery.
- •For vr hardware developers: Design VR headsets and controllers optimized for multi-view spatial perception to enhance teleoperation of delicate robotic systems.
Authors
Chang Liu, Chenhao Yu, Honghao Zhao, Hao Ding, Haochen Wei, Adnan Munawar, Mathias Unberath, Peter Kazanzides
Abstract
Current robot-assisted minimally-invasive surgery (RMIS) platforms provide a fixed console for the surgeon to view stereo endoscopic images and teleoperate instruments inside the patient. Several researchers have proposed the use of a head-mounted display (HMD) as a portable console, with video pass-through rendering of the endoscope images which, like the fixed console, restricts the operator to a single endoscopic viewpoint and limits depth perception. We present a digital twin-driven virtual reality (VR) teleoperation platform, where the digital twin is created from markerless perception of the surgical environment and streamed for display on the HMD. This overcomes the limitations of video pass-through by providing multi-view rendering and natural motion-parallax cues, enabling decoupling of the user's hand posture from strict instrument alignment. The system utilizes VR hand controllers to increase the teleoperation workspace and to improve the robustness and stability of instrument control compared to the hand tracking approach adopted by most prior systems. A 15-participant user study on the da Vinci Research Kit (dVRK) shows that our VR platform significantly outperforms a state-of-the-art HoloLens 2 mixed reality baseline, reducing path length by 86% and jerk by 95%, while achieving depth perception confidence comparable to or exceeding the traditional console across all conditions.