Robotic surgery learns from tool mounted sensors without cameras
From Instrument-Mounted Demonstrations to In-Vivo Execution: Learning Bimanual Laparoscopic Appendectomy Without Robot-Collected Demonstrations
Robotics
Summary
Most surgeries using small tools lose the details of how the surgeon moves their instruments, keeping only video footage. The authors created sensors that attach directly to the surgical tools, tracking their position and movements without needing external cameras. They collected data during real surgeries on rabbits to teach a robot how to perform a complex surgery step by step. The robot then successfully completed the surgery in live animals using only this sensor data, showing that robots can learn directly from the surgeon's own instruments instead of relying on robot-recorded demonstrations.
What this means in practice
- •For surgical robot developers: Train surgical robots on real instrument motion data captured without external cameras to improve autonomous procedure performance.
- •For medical device manufacturers: Integrate instrument-mounted sensors to capture tool motion for robot training without complex external tracking systems.$Commercial implications: Enables new surgical robot training products using self-contained sensors that lower setup costs and improve data capture accuracy.
Authors
Dongho Yee, Juahn Oh, Jinseok Lee, Jiyul Lee, Yechan Seo, Seong Jeong, Minsung Kim, Seonho Shim, Younghoon Noh, Hyuk Choi, Youngbin Kong, Kyu Eun Lee, Hyoun-Joong Kong
Abstract
Most minimally invasive surgery is still performed with hand-held laparoscopic instruments, and the surgeon's instrument kinematics are lost when the operation ends; only the endoscope video is kept. This paper presents an end-to-end pipeline that captures this motion in the operating room and uses it to train a surgical robot policy, validated on live animals. We introduce a surgical instrument-state logger that mounts on the shaft of a standard laparoscopic instrument and recovers its pose and jaw state from an inertial sensor, a time-of-flight sensor and a Hall sensor, with no external camera or tracker. A data pipeline measures the latency of every sensor channel against a robot ground truth and aligns the channels before forming observation-action pairs. On these demonstrations we train a diffusion policy with a fine-tuned DINOv3 backbone, selecting its design by closed-loop rollouts in a physics simulator reconstructed from depth maps of an ex-vivo rabbit appendix. The policy is then retrained on 849 in-vivo demonstrations from four live rabbits and deployed on four additional live rabbits with electrosurgery armed. With the surgeon selecting the surgical phase, the policy completed the appendectomy in three of the four animals. The results show that demonstrations recorded from a surgeon's own instruments are sufficient to train, select and deploy a bimanual surgical policy in vivo. The robot serves only as the timing reference for sensor calibration and as the executor, and collects no demonstrations. Both demonstration corpora are released to support future surgical robot learning research.