Robot takes surgical tool from surgeon without real training
SimRetract: Zero-Shot Sim-to-Real Robotic Assistance for Retractor Take-Over in Open Surgery
Robotics
Summary
Taking tools from a surgeon during open surgery is tricky because robots need careful training. This paper shows a robot can learn to take a retractor—a tool that holds open a surgical area—using only computer simulations and then perform the task in real life without extra training. The robot watches masked instrument images and understands how to grasp the tool, even when things aren’t perfect. The authors’ method works well on new retractors, suggesting it could handle similar tools too.
What this means in practice
- •For surgical robotics teams: Enable robot arms to take surgical retractors from surgeons smoothly without needing real-world training data.
- •For industrial robot developers: Design robot grasping policies that transfer from simulation to real tasks without human data or fine-tuning.
Authors
Xucheng Wang, Zhizhou Yang, Sung Eun Kim, Romain Hardy, Michael P. Sampson, Aakash Reddy, Ankit Pal, Xiaoman Zhang, Luyang Luo, Pranav Rajpurkar
Abstract
We demonstrate the first zero-shot sim-to-real transfer for a surgeon-robot collaboration task in open surgery: retractor take-over. A robot arm takes a retractor from the surgeon's hand and preserves retraction at handoff, using a policy trained only on simulated demonstrations, without human demonstrations or real-world fine-tuning. Imitation learning from real demonstrations is limited by collection and labeling costs, and by performance degradation when the environment changes. Our framework, SimRetract, generates simulated demonstrations designed for transfer: the policy observes masked instrument images and grasp geometry, learns to recover from off-nominal states, and acts with margin for execution drift. Deployed zero-shot, it grasps the retractor in 90% of 50 surgeon-defined phantom presentations and preserves retraction at handoff in 78%. Results on unseen retractors suggest potential for generalization to instruments with similar grasp geometry. Across ten policy configurations and 1,200 simulated and real trials, we localize the sim-to-real gap to components: in simulation, grasp rates across ablations span 80-96%, while on the real robot removing the instrument mask alone drops performance to 22% grasped and 12% held. Transfer-oriented design thus makes zero-shot sim-to-real transfer feasible for a surgical assistance task without human demonstrations. Code is available here: https://anonymous.4open.science/r/simretract-58D2.