Sea lm enables smart glasses to locate sounds and transcribe speech
SEA-LM: Egocentric Spatial Audio Understanding for Wearable Microphone Arrays
SoundArtificial Intelligence
Summary
People wearing smart glasses often hear sounds from many directions at once, which can be hard to understand. The authors created a system called SEA-LM that helps smart glasses figure out where sounds are coming from and what is being said, even when multiple people talk at the same time. They trained the system using recordings from various microphone setups on glasses and tested it on many configurations. This improves how well the glasses can separate overlapping voices and understand them more accurately.
What this means in practice
- •For wearable device developers: Create smart glasses that understand where sounds originate and transcribe speech in noisy, multi-speaker environments using flexible microphone arrays.$Commercial implications: This paper enables advanced spatial audio processing in wearable devices, improving user experience for consumers and professionals through more accurate sound understanding.
- •For consumer electronics teams: Improve hearing aids and augmented reality headsets with spatial audio awareness that enhances sound source separation and transcription accuracy.
Authors
Sonal Kumar, Sinan Hersek, Artem Dementyev, Mengzhen Pan, Ishan Chatterjee, Anurag Kumar, Ramani Duraiswami, Dinesh Manocha, Andrea Colaco
Abstract
Embodied, ego-centric intelligence fundamentally requires the ability to comprehend spatial audio within complex environments. While large audio-language models excel at mono-channel reasoning, they lack spatial awareness, discarding critical spatial cues that enable sound localization and that can improve the disentanglement of overlapping sound sources. To address this, we present SEA-LM, a Spatial Audio Understanding model. First, we introduce FOACODER, a layout-flexible spatial audio encoder trained on source localization and ego-centric voice activity detection objectives to encode First Order Ambisonics derived from variable-count, variable-position smart-glasses arrays via beamforming. We train a Multimodal Large Language Model (MLLM) to understand these spatial audio embeddings through a two-stage curriculum spanning six tasks, including sound localization and spatially selective transcription in settings with multiple speakers and overlapping sounds. To prevent the transcription outputs from dominating the next token prediction loss and overwhelming the direction predictions, we introduce a Spatio-temporal Weighted Cross-Entropy Loss. On our evaluation set, SEA-LM achieves lower azimuth and elevation MAE, higher temporal IoU, lower external-source hallucination and missing-source rates, and lower WER on most transcription tasks than compared baselines, while remaining robust across 1,211 smart-glasses array configurations with 4 to 9 microphones.