Automated endoscope control responds to surgeons using language

Language-Conditioned Behavioral Cloning for Autonomous Endoscope Guidance

Robotics

Summary

During surgeries with small cuts, doctors use tiny cameras called endoscopes to see inside the body. Controlling these cameras is hard because the doctor’s wishes about where to look change all the time, and it’s not always clear just from the video. The authors taught a computer to move the camera by listening to different spoken commands, like focusing or following a tool tip. Their tests show the computer can successfully follow many commands, proving this kind of language-guided camera control is possible.

What this means in practice

  • For surgical robotics teams: Implement language-guided camera control systems for endoscopes to reduce the need for manual operation during surgery.
  • For robotic control engineers: Develop multisensory robotic control policies that respond to spoken commands as well as visual input in other medical devices.

Authors

Ariel Rodriguez, Lorenzo Mazza, Ortrun Hellig, Alejandra Toscano, Rayan Younis, Stefanie Speidel

Abstract

Automated endoscope guidance reduces the need for continuous manual camera control during minimally invasive surgery. However, the desired camera motion depends on the surgeon's immediate intentions and cannot always be inferred from the visual operative scene alone. We propose to automate the task using language-conditioned behavioral cloning; we define 6 clinically relevant guidance behaviors and collect 235 subtask demonstrations. In preliminary phantom experiments, our policy achieves 100% success on focus, overview, and stop, 80% on search-tip and follow-tip behaviors, and 50% on horizon leveling. These results demonstrate the feasibility of learning heterogeneous surgeon-requested camera behaviors with a single visuomotor policy, while the observed failure modes motivate future work on adding temporal context for more demanding guidance tasks.