Speaker memory system improves multi-user voice assistant accuracy
From Voiceprints to User Entities: Online Speaker-Memory Construction for Multi-User Spoken Agents
SoundArtificial Intelligence
Summary
Spoken agents need to recognize who is talking to give personalized answers, but many systems assume they always know the speaker correctly. The authors show that a person’s voice is just a clue, not the actual identity, so the system should build a user profile over time as voice data changes. They created EvoSpeakerMem, a way to keep track of speakers and update their info smartly, making question answering better without confusing users. They also made a test called LoCoMo-ID to check how well systems handle multiple speakers and changing conditions.
What this means in practice
- •For voice assistant developers: Improve how multi-user voice assistants recognize and remember speakers over time despite varying recording conditions.
- •For call center software teams: Enhance speaker identification accuracy during long calls involving multiple participants to personalize service responses correctly.
Authors
Yucong Zhang, Cancan Li, Bo Xiao, Ke Zeng, Ming Li
Abstract
Personalized spoken agents must identify speakers before accessing user-specific memory, yet memory benchmarks typically assume correctly attributed histories. We argue that voiceprints are observations of identity, not identities themselves, and formulate online speaker-memory construction to maintain persistent user entities as acoustic evidence evolves. We introduce EvoSpeakerMem, a training-free layer for causal speaker routing, selective acoustic-state updates, and identity-partitioned memory writes and retrieval through a replaceable text-memory backend. We also construct LoCoMo-ID, a spoken-memory benchmark with hidden speaker identities, speaker-sensitive queries, simulated recording-condition shifts, and four-speaker identity-confusion tests. On longitudinal SMIIP-TV, genuine-probe rejection under a frozen near-enrollment threshold rises from 29.8% to 55.6% over approximately one year. On LoCoMo-ID speaker-sensitive queries, gated updating improves identity-gated QA over fixed representations by 3.86 and 11.62 percentage points under clean and shifted conditions, respectively, with no wrong-user authorizations. The advantage persists in the four-speaker test. Yet reducing fragmentation and improving QA can coexist with incorrect cross-user links, revealing a fragmentation--contamination trade-off that QA alone cannot capture.