Surgical vision models improve at understanding actions over time
STSG-VQA: Evidence-Grounded Temporal Question Answering from Surgical Spatio-Temporal Scene Graphs
Computer Vision and Pattern Recognition
Summary
Current computer systems that analyze surgical videos struggle to understand how events change and connect over time because they mostly look at single video frames. The researchers enhanced these systems by creating multi-level timelines that track objects, actions, and stages throughout surgeries. They developed a new test set with questions tied to these timelines, so the system’s answers can be checked and traced back. Training models on this new information helped them get much better at answering questions about surgical processes as they unfold, rather than just isolated moments.
surgical vision-language modelstemporal reasoningscene graphsspatio-temporal graphsworkflow modelingquestion answeringvideo analysisevidence grounding
Authors
Jing Li, Duygu Sarikaya
Abstract
Despite recent advances in surgical vision-language models (VLMs), temporal reasoning remains limited because existing supervision is largely frame-centric. Frame-level scene graphs (SGs) have proven effective in providing structured representations of surgical environments but do not explicitly model the dynamics of surgical workflows. To explicitly model how surgical states evolve across time, we introduce a multi-level structured temporal supervision methodology that augments frame-level surgical SGs with object-level continuity, event-level interaction continuity, and procedure-level connectivity. We then execute temporal queries over the resulting spatio-temporal scene graphs (STSGs) to generate evidence-grounded question-answer pairs, which together form the STSG-VQA benchmark. Each question is linked to the temporal interval and STSG evidence used to derive its reference answer, enabling traceable verification. The benchmark contains 18,458 question-answer pairs across seven temporal categories. Fine-tuning Qwen3-VL-4B and Hulu-Med-4B with STSG-derived supervision improves question-level micro accuracy by 24.39 and 19.56 percentage points over their zero-shot baselines and by 16.50 and 14.25 points over static scene-graph supervision, respectively. These gains span all temporal categories, indicating that STSG-derived supervision helps surgical VLMs reason over temporally grounded interactions rather than isolated frames. The code and dataset will be made publicly available upon acceptance.