SurgGraph improves surgical video understanding with detailed scene graphs
SurgGraph: Quantitative Laparoscopic Video Understanding via Geometry-Grounded Scene Graphs
Human-Computer Interaction
Summary
Surgical videos are important teaching tools, but computers find it hard to understand the details of what's happening in them. The authors created SurgGraph, a system that looks at surgical videos and breaks down the scene into precise relationships, like how tools touch or cover tissues, and measures these over time. This makes it easier to analyze and learn from surgeries. They also built SurgGraphQA, a tool that helps medical students by showing examples and explanations, which helped users learn better in tests.
What this means in practice
- •For surgical training teams: Provide detailed automatic analysis of laparoscopic videos to improve trainee feedback and skill assessment.
- •For medical device developers: Develop intelligent surgical support tools that detect and quantify instrument interactions and anatomical context.
Authors
Jingying Wang, Rosiana Natalie, Marquise D Singleterry, Filippos Bellos, Brian George, Gurjit Sandhu, Jason J Corso, Anhong Guo, Vitaliy Popov, Xu Wang
Abstract
Surgical videos are a primary resource for teaching trainees anatomy, tool usage, and procedural skills. Yet learning from them at scale requires systems that understand surgical scenes. Existing approaches fall short: vision-language models lack fine-grained domain reasoning, task-specific models do not generalize, and prior scene graphs omit clinically meaningful detail. We present SurgGraph, a training-free pipeline that generates quantitative scene graphs from surgical videos. Operating on segmentation masks and depth maps, SurgGraph encodes each clinically meaningful relation (attachment, occlusion, separation, tool actions) as a <subject, verb, object, value> tuple whose numeric value quantifies the relation's extent over time. Technical evaluations show more precise scene understanding than state-of-the-art surgical VLM baselines. We then build SurgGraphQA, a proof-of-concept learning application that retrieves meaningful and boundary-case exemplars and generates visual explanations and feedback. A study with 17 medical students and 2 resident surgeons shows significant learning gains, demonstrating its educational value.