Surgical Re-enactment for Operating Room Workflow Datasets
2026-07-27 • Robotics
RoboticsComputer Vision and Pattern Recognition
AI summaryⓘ
The authors explain that to create smart operating rooms using new technologies like surgical robots, they need detailed data showing what everyone in the room is doing during surgery. Since collecting this data in real surgeries is very hard because of ethical rules and space limits, they designed a way to re-enact entire surgeries in a reconstructed operating room. This method allows them to record repeatable and well-labeled data to help train computer models that understand surgical actions. Their approach was developed specifically for eye surgeries assisted by robots but can be adapted to other types of surgeries.
surgical workflowoperating room (OR)robot-assisted surgeryactivity recognitiondataset annotationsurgical process modelingophthalmic surgeryre-enactmentpost-take debriefingscene graph
Authors
Jana Nina Friedrich, Andrea Karin Maria Ross, Angelo Henriques, Mario Peter Martin Weisser, Ling Zhang, Mohammad Ali Nasseri
Abstract
The introduction of new technologies, such as surgical robots, is driving the vision of a connected, smart operating room (OR). However, realizing this vision requires a deep understanding of surgical workflows, which relies on realistic datasets capturing the actions of all OR personnel from both full room and surgical field perspectives. Acquiring such data in real ORs is prohibitively challenging due to factors such as ethics committee approvals, limited space for camera installation, and sterility regulations preventing the use of tracking markers. We present a step-by-step methodology for re-enacting complete surgical procedures in a reconstructed OR. This approach enables the creation of repeatable and annotatable workflow datasets for training activity recognition models, generating scene graphs, and formalizing surgical process models. Developed for robot-assisted ophthalmic surgery, our methodology combines expert consultation, structured workflow formalization, OR reconstruction, role-based training, real OR observation, and iterative recording with post-take debriefing. We provide concrete recommendations to allow other research groups to seamlessly adopt this methodology for their own surgical domains.