SurgVIL: Scaling Surgical Robot Imitation Learning with Open-source Surgical Videos
2026-08-17 • Robotics
Robotics
AI summaryⓘ
The authors address a challenge in teaching surgical robots to perform tasks by imitation: real surgical videos rarely have detailed robot movement data, while controlled research data lack realistic visuals. They created SurgVIL, a method that combines accurate robot movement data from controlled phantom setups with lots of open-source surgical videos. To handle videos without detailed robot data, they estimate approximate robot motions to guide learning. Their tests on robot tasks showed that using these combined videos helps robots perform better on real tissue and unfamiliar situations.
surgical robot autonomyimitation learningphantom datakinematicsda Vinci robotneedle pick-upcholecystectomyopen-source surgical videospolicy learningweak supervision
Authors
Xinhao Chen, JuoTung Chen, Nigel Nelson, Antony Goldenberg, Jesse Haworth, Sean D. Huver, Axel Krieger
Abstract
Learning-based surgical robot autonomy requires large-scale demonstrations with synchronized videos and robot actions, but such data are exceedingly rare in clinical or realistic tissue settings because robot kinematics are typically inaccessible outside controlled research systems. In contrast, phantom data collected on research platforms provide accurate action labels but lack the visual diversity of real tissue. We propose SurgVIL, a framework for scaling surgical robot imitation learning using open-source surgical videos. SurgVIL combines kinematically labeled phantom robot demonstrations with surgical videos from open-source datasets and online sources for policy learning. Since these videos lack robot motion labels, we estimate approximate kinematics as weak supervision. We evaluate SurgVIL on two da Vinci robot tasks: needle pick-up and cholecystectomy cutting. Across ACT, $π_0$, and GR00T-H backbones, adding surgical videos substantially improves generalization to real-tissue and out-of-distribution settings, suggesting a scalable path from phantom training toward generalizable surgical robot policies.