Surgical point tracking improves accuracy using sparse endpoints and video data
STEP: Surgical Point Tracking from Sparse Endpoints and Pseudo-trajectories
Computer Vision and Pattern Recognition
Summary
Tracking points on surgical tools or tissue during operations is hard because the tissue moves and parts get blocked from view. The authors created a method called STEP that learns how points move using short, sparse annotations and video instead of relying on costly, dense tracking labels. STEP improves accuracy by combining physical endpoint information with video cues to maintain consistent tracking even when things shift or get hidden. This method can better track points in surgical videos without needing detailed manual labeling of every frame.
What this means in practice
- •For surgical software developers: Develop tools that accurately track surgical points from standard video with minimal manual labeling, improving workflow in image-guided surgery.
- •For robotic surgery teams: Use improved surgical point tracking for better motion understanding and control in robotic-assisted procedures without extensive new data collection.
Authors
Bing Fan, Anwesa Choudhuri, Zhongpai Gao, Meng Zheng, Benjamin Planche, Terrence Chen, Heng Fan, Ziyan Wu
Abstract
Surgical point tracking requires persistent correspondence through tissue deformation and occlusion, yet dense trajectory annotations are costly to obtain. We present STEP, a two-stage framework that learns from surgical video and sparse physical endpoints. First, a frozen offline teacher generates candidate trajectories, and temporal reversibility and stereo consistency select dense supervision for a causal student. Second, a robust physical endpoint loss anchors terminal correspondence, while a round-trip loss constrains temporal consistency. A gradient-free prefix exposes training to accumulated drift, while a final differentiable window bounds activation memory. The trained tracker processes a single RGB stream with fixed weights. In STIR-to-VL-SurgPT transfer, temporal-stereo union selection reduces endpoint error by 7.0% relative to unfiltered trajectories. On a post-hoc diagnostic of 29 VL-SurgPT clips excluded from the endpoint-training manifest, STEP reduces endpoint error from 31.52 to 16.80 pixels relative to dense-label fine-tuning. These results support combining curated trajectories with physical observations to train surgical point trackers.