Robotic endoscopy uses language to find hidden targets reliably
EndoVLA-OOV: Language-Conditioned Out-of-View Target Acquisition in Robotic Endoscopy
Robotics
Summary
Endoscopy is a medical process where a tiny camera is guided inside the body to look for issues. Sometimes doctors must bring a suspicious area they saw earlier back into view for a closer look, which is hard because the camera moves inside a tight, flexible space. The authors created EndoVLA-OOV, a system that understands language instructions and uses previous and current images to guide the robotic camera back to these hidden targets. Their system was tested successfully across different settings, including low light and new environments, showing it can learn from more examples to improve accuracy.
What this means in practice
- •For medical device teams: Design robotic endoscopy systems that navigate to targets based on natural language descriptions within flexible anatomical spaces.$Commercial implications: Enables creation of smarter robotic endoscopy products with improved target reacquisition using language-guided control, enhancing clinical workflows.
- •For surgical robotics engineers: Develop vision-language-action models that adapt to different robot embodiments and unseen physical environments for precise instrument guidance.
Authors
Chi Kit Ng, Yidong Zhang, Tianchun Wu, Zhiqing Tang, Hanlin Zhao, Siu Hing Lui, Tao Yang, Jingsong Lin, Huxin Gao, Yeung Trevor, Raymond Tang, Hongliang Ren
Abstract
The scan-then-revisit workflow of upper gastrointestinal endoscopy requires bringing a previously observed suspicious region back into view for closer inspection. Acquiring such targets requires translating anatomical and appearance descriptions into camera motions from partial observations of a confined, deformable environment. We present EndoVLA-OOV, a language-conditioned vision-language-action policy for out-of-view target acquisition in robotic endoscopy. The policy adapts a pretrained backbone to continuous two-degree-of-freedom distal bending commands and combines initial, intermediate, and current images to guide search. To support policy learning, we provide an open-source VLA dataset containing 1,050 episodes across six tasks and approximately 262k observation-action pairs. Physical evaluation spans wet-lens and low-light conditions, two unseen stomach phantoms, and cross-embodiment transfer to an unseen ex-vivo environment. In the cross-embodiment evaluation, the full-data policy achieves 93.3% target acquisition success (28/30). In a controlled training-data comparison, expanding the corpus from 312 to 1,050 episodes increases acquisition success from 40.0% to 90.0% under the same model, training, and test configuration. Instruction and observation interventions further support the use of language and temporal context in the evaluated acquisition tasks. These results support language-guided endoscopic revisitation across the evaluated environments and robot embodiments and demonstrate the benefit of expanded task-specific demonstrations.