Large audio language model aids smartglasses speech transcription
Frontend-Assisted Large Audio-Language Models for SLT2026 SmartGlasses Challenge
Sound
Summary
Understanding speech when two people talk over each other or when there’s background noise is really hard for devices like smartglasses. The authors use a special combined audio and language model called SoulX-Transcriber that listens to full conversations and writes down who said what and when. They also try extra audio processing to separate voices better before transcription. Their final system worked well on tricky test sets with overlapping speech and noise.
What this means in practice
- •For wearable device developers: Build smartglasses that can accurately transcribe and attribute speech in noisy, multi-speaker conversations.$Commercial implications: Enables sale of smartglasses with advanced real-time transcription features for consumer or enterprise use.
- •For hearing aid manufacturers: Improve hearing aids with better voice separation and transcription to assist users in noisy two-person conversations.
Authors
Yushi Wang, Zeyan Song, Xiaobin Rong, Chengwei Liu, Jun Gao, Jing Lu
Abstract
This paper describes our systems for the Track 1 time-stamped speaker-attributed ASR task of the SmartGlasses Challenge. To handle overlapping speech, background noise and far-field conditions in two-person conversations, we employ SoulX-Transcriber, a multimodal large language model that takes full-length audio as input and generates structured transcriptions. In addition, we investigate the effect of speech frontend processing, including a rule-based Guided Source Separation method and a data-driven speech separation model, both of which improve performance on the validation set. Considering inference stability, our final submission uses the fine-tuned SoulX-Transcriber system, achieving a tcpCER of 6.22 on the test set.