Surgcast predicts surgical tool movement to control video generation
SurgCast: Action-Conditioned Future Skeletons for Controllable Surgical World Models
Computer Vision and Pattern Recognition
Summary
In surgery, it’s important to predict how robotic tools will move next to create realistic video simulations. The authors introduce SurgCast, a system that forecasts future positions of surgical tool skeletons based on current state and planned actions. SurgCast then uses these predictions to generate more accurate and consistent surgical videos. Their tests show that predicting future tool motion helps improve the quality and reliability of these video models.
What this means in practice
- •For surgical robotics teams: Create realistic video simulations of robotic instruments to improve planning and control during surgeries.
- •For medical device developers: Develop advanced surgical video systems that predict instrument motions for enhanced training and verification.$Commercial implications: Enables new surgical simulation and monitoring tools with better video accuracy, attractive for training and clinical use.
Authors
Wanhao Liu, Rulin Zhou, Liangjing Shao, Zhaocheng Lin, Dongyue Li, Jinsong Lin, Guoheng Ma, Hangheng Yang, Zhiqing Tang, Jing Chen, Panshuo Li, Hongliang Ren
Abstract
Surgical world models must turn prospective robot actions into visually faithful, structurally consistent instrument motion. We introduce SurgCast, which predicts future instrument skeletons from current structure, robot state, and future actions, then uses the forecast to control video generation. Dual-path injection combines skeleton geometry with derived, instrument-specific motion descriptors through geometric residuals and conditioning modulation. Both paths complement native action conditioning and are retained in a four-step causal student. On SutureBot and SRTH-Porcine-Cholei, the teacher and student improve all seven reported quality and instrument-consistency metrics over their respective direct-action baselines; teacher FVD decreases by 37.6% and 67.3%. Ablations show that future actions improve skeleton prediction and that combining both injection paths performs best across all six metrics under predicted skeletons. These results support using predicted articulated structure and its motion semantics as a control interface for surgical video prediction. Project videos and examples are available at https://surgcast.netlify.app/.