Speaker temporal memory improves multi speaker speech recognition accuracy
STAM-ASR: Speaker-Temporal Anchoring with Memory for Multi-Speaker ASR
Sound
Summary
Understanding who is speaking and when they talk is hard in conversations where people speak over each other and come back later. The authors created a method called STAM-ASR that helps speech recognition systems figure out speaker identity and timing without needing separate steps to separate voices. It uses memory to remember who spoke earlier and builds on existing speech AI models. Tests on multiple real-world conversation types show that remembering speaker info and timing helps, but accurately tracking speakers over time remains tough.
What this means in practice
- •For conference transcription teams: Transcribe meetings with overlapping speakers while identifying who spoke when without separate diarization tools.
- •For call center operators: Improve speaker-attributed transcription for multi-person calls, enabling clearer records without costly speaker separation systems.
- •For video editing software developers: Automatically tag speakers over time in multi-person conversation recordings to assist video indexing and editing workflows.$Commercial implications: Enables new speaker-aware transcription features that can be sold as value-added tools in video production software.
Authors
Victor Tolulope Olufemi, Syeda Faiza Ahmed Sara, Shammur Absar Chowdhury
Abstract
Natural conversations make both speech recognition and speaker attribution challenging for ASR, as speakers take turns, overlap, and reappear over time. We propose STAM-ASR, Speaker-Temporal Anchoring with Memory, a lightweight framework that extends an already pretrained AudioLLM for multi-speaker ASR. Without relying on an external diarization system, STAM-ASR learns speaker activity and speaker-aware representations directly from intermediate AudioLLM features. Hence providing explicit who and when cues to modulate the AudioLLM's semantic representation without explicit speech separation. STAM-ASR further maintains fixed-size speaker and conversational memories to carry complementary context across turns. We evaluate STAM-ASR on AMI, ICSI, LibriCSS, and NOTSOFAR-1 across close-talk, far-field, overlapping, and cross-domain conditions. Our reported results shows that speaker-temporal conditioning and memory provide complementary benefits, while the gap between reference and predicted speaker activity identifies robust speaker tracking as a key remaining challenge.