Vision system estimates surgical tissue forces using 3d reconstruction
Object-Centered Reconstruction for Vision-Based 3D Force Estimation
Robotics
Summary
Too much force during robotic colorectal surgery can harm tissue and cause complications. The authors developed a way to estimate how hard surgical tools press on tissue just by looking at stereo video from an endoscope. They reconstruct the tissue shape in 3D and track it carefully to predict the 3D forces using a neural network. Their method works well on models and real tissue, and it could help surgical robots without force sensors measure forces safely.
What this means in practice
- •For surgical device manufacturers: Implement vision-based force estimation on robotic surgery platforms lacking force sensors to improve safety and reduce tissue damage.$Commercial implications: Enables selling enhanced surgical robot systems with non-contact force sensing features to hospitals and surgical centers.
- •For medical software developers: Build software that uses stereo endoscopic video to estimate tissue interaction forces, enhancing surgical tool feedback in existing robotic platforms.
Authors
Zhonghao Zhang, Mingyeung Wu, Hao Yang, Ayberk Acar, Alan Kuntz, Jie Ying Wu
Abstract
Excessive force may damage tissue and increase the risk of anastomotic leakage in robotic colorectal surgery. Although the da Vinci 5 provides force sensing, this capability is unavailable on earlier da Vinci systems and many other surgical robotic platforms. In this work, we present a vision-based pipeline for estimating 3D interaction forces from soft-tissue deformation in stereo endoscopic video. We dynamically reconstruct the tissue point cloud in an object-centered coordinate frame, track tissue points with geometric constraints, and predict the 3D force vector with a neural network. We progressively evaluate the pipeline on rubber-glove phantoms, ex vivo porcine colons, and in vivo colorectal surgical video sequences. Under varying tissue orientations and positions within the endoscopic view, as well as different camera viewpoints, the proposed method achieves average root mean square error (RMSEs) of 0.77 N and 1.30 N on the phantom and porcine colon, respectively. Compared with the camera-frame representation, the object-centered representation reduces average RMSE by 51.3% and 56.7%, while geometry-constrained tracking reduces RMSE by 19.8% and 25.3% compared with CoTracker. We further qualitatively demonstrate the feasibility of vision-based force estimation on an in vivo colorectal surgical sequence, as a step toward clinical translation of vision-based, sensorless force estimation.