Surgical video question answering improves with time-aware scene graphs
STSG-VQA: Evidence-Grounded Temporal Question Answering from Surgical Spatio-Temporal Scene Graphs
Computer Vision and Pattern Recognition
Summary
Understanding how surgical actions happen over time is hard because most computer models look at single video frames, not the full sequence. The authors created a new way to represent surgical videos that links objects and events across time, forming detailed timelines called spatio-temporal scene graphs. They then make questions about these timelines to test how well models understand surgical procedures. Their experiments show that training models with this new timeline information helps them answer more questions correctly about what happens during surgery.
What this means in practice
- •For medical ai developers: Improve surgical assistant AI systems to better understand and answer questions about surgical workflows over time using temporal graph data.
- •For medical video annotation teams: Create more accurate and temporally grounded annotations for surgical video datasets by using multi-level temporal scene graphs.
Authors
Jing Li, Duygu Sarikaya
Abstract
Despite recent advances in surgical vision-language models (VLMs), temporal reasoning remains limited because existing supervision is largely frame-centric. Frame-level scene graphs (SGs) have proven effective in providing structured representations of surgical environments but do not explicitly model the dynamics of surgical workflows. To explicitly model how surgical states evolve across time, we introduce a multi-level structured temporal supervision methodology that augments frame-level surgical SGs with object-level continuity, event-level interaction continuity, and procedure-level connectivity. We then execute temporal queries over the resulting spatio-temporal scene graphs (STSGs) to generate evidence-grounded question-answer pairs, which together form the STSG-VQA benchmark. Each question is linked to the temporal interval and STSG evidence used to derive its reference answer, enabling traceable verification. The benchmark contains 18,458 question-answer pairs across seven temporal categories. Fine-tuning Qwen3-VL-4B and Hulu-Med-4B with STSG-derived supervision improves question-level micro accuracy by 24.39 and 19.56 percentage points over their zero-shot baselines and by 16.50 and 14.25 points over static scene-graph supervision, respectively. These gains span all temporal categories, indicating that STSG-derived supervision helps surgical VLMs reason over temporally grounded interactions rather than isolated frames. The code and dataset will be made publicly available upon acceptance.