Sounding surgical tools located using single microphone and camera
Sounding Surgical Instrument Localisation through Audio-Visual Correspondence
SoundComputer Vision and Pattern Recognition
Summary
Understanding what is happening during surgery is important for smarter medical systems. Most current technology uses cameras only, but sounds can also give clues about surgical actions. The authors show it is possible to locate an active surgical tool using just one microphone and one camera by teaching a computer to match the tool's look with its sound, without needing exact position labels. Their method can tell which tool is being used even when two tools are visible and can ignore background noises. This approach could help detect surgical activity more easily without complex setups.
What this means in practice
- •For operating room technology teams: Develop systems to detect and locate active surgical instruments during operations using minimal audio-visual equipment and no instrument position labeling.
- •For manufacturers of medical training simulators: Integrate audio-visual localisation to provide real-time feedback on tool usage without manual annotations during simulated surgical training.$Commercial implications: Enables enhanced simulator features that automatically detect and localize tools for training feedback, improving product value.
Authors
Dominic Behling, Jonas Hein, Matthias Seibold
Abstract
Surgical scene understanding is a prerequisite for context-aware and intelligent systems in the operating room. Current methods rely mostly on visual data, while acoustic sensing has shown potential to provide complementary information on surgical activity. First approaches for the localisation of surgical activity relied on microphone arrays and acoustic beamforming. This work investigates whether the active instrument can be localized from a single microphone and a camera by learning the correspondence between the appearance and the sound of surgical instruments, without annotating its position. We adapt weakly supervised audio-visual localisation, which learns from the co-occurrence of image and sound, to the surgical setting. We record simulated surgical actions, sawing, drilling, and chiseling, and train the model on these recordings without any spatial annotation to produce a heatmap over the sounding tool. We extend the training objective by two terms: a silence term, which suppresses the output for silence and background noise, and a composite term, which trains the model to select the sounding one of two visible instruments. The model localizes the active instrument on an independent corpus recorded with a different camera and microphone, in which instrument positions were obtained by optical tracking. It separates tool activity from silence and background noise and can attribute sound correctly when two tools are present. Audio-visual correspondence learned from in-domain recordings connects the sound of a surgical action to the instrument that produces it, using a single microphone and a camera. This proof of concept introduces a novel paradigm that enables the detection and localisation of surgical activity without manual annotations.