surgical training teams

Abstract: Objective assessment of robotic surgery uses instrument kinematics, which must be reconstructed when only video is available. We introduce a kinematic reconstruction network for estimating instrument position, orientation and jaw angle from monocular video. Our visual representation combines global attention pooling of frozen DINOv3 features with local pooling at instrument landmarks from fine-tuned SAM 3.1 masks. Our shared Transformer encoder and temporal convolutional heads integrate this representation with mask geometry, monocular depth and visual state estimates from arm-specific multilayer regression networks. Our position branch predicts displacement magnitude and direction separately to preserve traveled distance. We fit trajectories to predicted state observations and motion increments by differentiable weighted least squares, expressing quaternion observations relative to cumulative predicted rotations to obtain a quadratic orientation objective. We evaluate reconstruction across 2,802 Open-H episodes. Compared with LiveMAE on the main Open-H benchmark, our method reduces path-length mean absolute error from 0.45 to 0.34\,cm and increases temporal mean average precision for motion segmentation from 44.54\% to 54.44\%.

Abstract: Surgical videos are a primary resource for teaching trainees anatomy, tool usage, and procedural skills. Yet learning from them at scale requires systems that understand surgical scenes. Existing approaches fall short: vision-language models lack fine-grained domain reasoning, task-specific models do not generalize, and prior scene graphs omit clinically meaningful detail. We present SurgGraph, a training-free pipeline that generates quantitative scene graphs from surgical videos. Operating on segmentation masks and depth maps, SurgGraph encodes each clinically meaningful relation (attachment, occlusion, separation, tool actions) as a <subject, verb, object, value> tuple whose numeric value quantifies the relation's extent over time. Technical evaluations show more precise scene understanding than state-of-the-art surgical VLM baselines. We then build SurgGraphQA, a proof-of-concept learning application that retrieves meaningful and boundary-case exemplars and generates visual explanations and feedback. A study with 17 medical students and 2 resident surgeons shows significant learning gains, demonstrating its educational value.

Abstract: Accurate gaze tracking is essential for understanding surgeons' visual attention and cognitive processes during laparoscopic surgery, yet wearable eye trackers produce large errors systematically correlated with ground-truth gaze locations, as demonstrated in Study 1. We introduce SurgGaze, an implicit calibration method that corrects these errors using high-confidence surgical moments. Building on evidence that surgeons' gaze converges near the tool-tissue contact point (TTCP) during dissection, SurgGaze uses TTCP as a surrogate for true gaze to construct training pairs. We evaluate SurgGaze in a simulated operating room trial and an authentic operating room case study. In simulation, SurgGaze reduced gaze estimation error by 40.6%, significantly outperforming conventional 9-point explicit calibration. The case study showed that these moments provide reliable training data and that calibrated gaze improves interpretation of surgeons' attention beyond numeric error reduction. These findings demonstrate that structured behavioral signals can enable implicit calibration for gaze tracking in complex real-world settings.