Robotic surgery instrument movement tracked from single video
Surgical Kinematics from Monocular Video with Learned Articulated Motion Constraints
Computer Vision and Pattern RecognitionRobotics
Summary
Measuring how surgical instruments move during robot-assisted operations helps experts assess performance, but often only video recordings are available. The authors created a computer model that can figure out the exact position, angle, and jaw movements of instruments from just one camera's view. They use clever visual features, attention techniques, and mathematical fitting to improve accuracy. Their approach worked better than previous methods on a large surgical video dataset.
What this means in practice
- •For surgical training teams: Analyze robotic surgery videos to reconstruct instrument movements without extra sensors for objective skill evaluation.
- •For medical imaging software developers: Incorporate monocular video kinematic reconstruction to enhance surgical navigation and monitoring tools with detailed instrument tracking.
Authors
Mehmet Kerem Turkcan, Soham Samal, Zoran Kostic
Abstract
Objective assessment of robotic surgery uses instrument kinematics, which must be reconstructed when only video is available. We introduce a kinematic reconstruction network for estimating instrument position, orientation and jaw angle from monocular video. Our visual representation combines global attention pooling of frozen DINOv3 features with local pooling at instrument landmarks from fine-tuned SAM 3.1 masks. Our shared Transformer encoder and temporal convolutional heads integrate this representation with mask geometry, monocular depth and visual state estimates from arm-specific multilayer regression networks. Our position branch predicts displacement magnitude and direction separately to preserve traveled distance. We fit trajectories to predicted state observations and motion increments by differentiable weighted least squares, expressing quaternion observations relative to cumulative predicted rotations to obtain a quadratic orientation objective. We evaluate reconstruction across 2,802 Open-H episodes. Compared with LiveMAE on the main Open-H benchmark, our method reduces path-length mean absolute error from 0.45 to 0.34\,cm and increases temporal mean average precision for motion segmentation from 44.54\% to 54.44\%.