SurgStream enables timely real-time surgical video understanding
SurgStream: A Workflow-Anchored Proactive Streaming Framework and Benchmark for Real-Time Surgical Video Understanding
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Understanding videos of surgeries in real time can help doctors by tracking the progress and preparing tools or warnings. The authors created SurgStream, a system that not only watches what’s happening but also remembers important past details and predicts what will happen soon. Their system uses a special way to organize memories based on surgical steps and learns when exactly to respond before important moments pass. They tested SurgStream with many examples of surgical videos and showed it works better than previous methods, especially when answering questions and alerting proactively.
What this means in practice
- •For surgical device developers: Integrate SurgStream to enhance surgical systems with real-time phase detection and tool usage memory for better intraoperative assistance.$Commercial implications: This enables commercial surgical support products that provide timely, intelligent alerts and documentation during operations.
- •For medical software engineers: Develop intraoperative monitoring tools that proactively answer surgical queries and alert staff before critical surgical deadlines.
Authors
Ruipeng Ma, Qing Xu, Jie Liang, Zhen Chen
Abstract
Surgical video understanding is essential for intraoperative assistance such as workflow monitoring, instrument preparation and procedural documentation. Delivering such assistance in real time requires a model to jointly perceive the ongoing scene, remember past surgical events, and respond before clinically relevant moments pass. Existing approaches provide limited support for jointly retaining procedural evidence and deciding when to respond in streaming surgical video. In this work, we propose SurgStream, a workflow-anchored proactive streaming framework for real-time surgical video understanding. We first introduce Workflow-Anchored Memory (WAM) to record structured instrument-action-target observations and fold them into phase-level summaries, providing compact and queryable surgical memory. Deadline-aware Transition Learning (DTL) is then devised to supervise response states within clinical windows, enabling timely responses before procedural deadlines. We further introduce SurgStream-Bench, comprising 11.7K streaming interactions across various tasks covering real-time perception, backward memory queries, forward proactive QA, proactive alerts and caption memory. Experiments on SurgStream-Bench show gains over state-of-the-art methods, especially in memory and forward proactive QA. The source code will be available.