EndoLive enables real-time video style conversion for surgical training
EndoLive: Real-Time Style Transfer for Endoscopic Endonasal Skull Base Surgical Video
Computer Vision and Pattern Recognition
Summary
Training surgeons on real human patients is risky, so they practice on cadavers first. But cadaver tissue looks different from living tissue, making it hard to recognize important anatomy during practice. The authors created EndoLive, a system that changes the look of cadaver videos to resemble live surgery videos in real-time. This helps surgeons better prepare by seeing cadaver anatomy as it would appear during actual surgery.
What this means in practice
- •For surgical trainers: Provide trainees with more realistic cadaver video visuals that appear like live surgery for improved surgical rehearsal.
- •For medical simulation developers: Integrate real-time cadaver-to-live video style transfer into surgical simulators to enhance visual authenticity.
Authors
Griffin Hurt, Calvin Brinkman
Abstract
Complex surgical procedures around critical anatomy, such as the endoscopic endonasal skull base surgery, requires significant practice and training on the part of the surgeon before they are allowed to perform the operation on a live patient. This training in typically done in cadaveric specimens, due to them containing the same critical structures as a living human. However, cadavers are not a perfect 1-to-1 substitute for a living patient. The dead and preserved tissues of a cadaver are colored completely differently than a living human, and -- without complex and expensive pumping systems -- do not bleed in the same way. As a result, identifying the critical pieces of anatomy that make this procedure so complex can be quite different in a live case than in a surgeon's cadaveric practice. This paper presents EndoLive, a framework for real-time style transfer between cadaveric endoscopic video and living human endoscopic video. Our method combines the ConStructS GAN model for realistic style transfer for surgical applications, with the HyPER-GAN model that can learn complex translations and perform them in real-time. We train EndoLive on unpaired cadaveric and live images taken from an endoscope, and test the trained model with cadaveric video, on a variety of devices. Experimental results demonstrate that EndoLive can perform cadaveric-to-live translation at speeds well above the minimum necessary for real-time, while maintaining semantic consistency of critical anatomical structures. Our source code is available at https://github.com/griffhurt/endolive.