StenoVLA-3D improves robotic endoscopy navigation through narrow passages
StenoVLA-3D: 3D-Aware Reasoning VLA for Navigation Through Gastrointestinal Stenoses
RoboticsComputer Vision and Pattern Recognition
Summary
Navigating tiny, narrow areas inside the body with a camera on a robot is very hard because the images often lack clear detail. The authors created a computer model called StenoVLA-3D that uses 3D information and remembers what it has seen over time to safely guide the robot and identify problematic spots. They also made a new dataset to train and test this approach. Their method performed better than existing ones both on recorded videos and physical robot tests in realistic models of the esophagus and colon.
What this means in practice
- •For medical device developers: Develop robotic endoscopes that navigate safely through narrow gastrointestinal regions using 3D-aware guidance and lesion reporting.$Commercial implications: This paper enables creation of advanced robotic endoscopy products that improve diagnostic navigation and lesion tracking inside patients.
- •For medical training centers: Use physical phantoms and the StenoVLA-3D system to train clinicians on navigating and detecting lesions in stenotic gastrointestinal pathways.
Authors
Tamima Tabassum, Yiming Huang, Tianchun Wu, Changjing Liu, Zhiqing Tang, Chikit Ng, Beilei Cui, Liangjing Shao, Jiewen Lai, Hongliang Ren
Abstract
Autonomous endoscopic navigation requires the policy model to predict actions from texture-poor monocular observations, make safe control decisions, and retain evidence of lesions after they leave the field of view. Existing vision-language-action (VLA) models primarily rely on visual appearance and short-term context, limiting geometric grounding and episode-level reporting. We introduce StenoVLA-3D, a 3D-aware VLA framework for navigating through stenotic regions. We integrate point-maps into the Cosmos-Reason 2 backbone through learned geometry-gated fusion, and also propose a temporal state branch to model traversal progress. Our reasoning-and-action backbone predicts grounded reasoning with actions, while dedicated heads estimate stenosis shape and generate the final lesion report. We further introduce EndoCausal, an episode-level dataset with lesion annotations, actions, and temporally grounded reasoning. On 40 held-out recorded test episodes, StenoVLA-3D reaches 95.2\% semantic accuracy and 83.4\% action accuracy. On the physical 3-DoF endoscope, it attains 88.9\% and 77.8\% task success in esophageal and colonic phantoms (36 trials each), substantially outperforming the evaluated baselines.