Persistent memory helps voice assistants remember users over time
PERSIST: Who-What-When Memory Across Sessions for Full-Duplex Spoken Dialogue
Artificial Intelligence
Summary
Voice assistants often talk to many users and need to remember details from previous conversations, like who said what and when. The authors created PERSIST, a system that keeps track of who spoke, what they said, and when across different sessions. PERSIST organizes past conversations into clear event records and matches new questions to the right past information quickly. The authors also made a test called SpokenTrace to check how well systems remember past talks and keep track of speakers and time. PERSIST performed well in these tests, showing it can improve voice assistants' memory and user understanding.
What this means in practice
- •For voice assistant developers: Improve multi-user voice assistants to remember who said what and when across conversations and respond more accurately.
- •For customer service platform engineers: Enhance call center systems to track customer interactions over time including speaker identity and conversation history.
Authors
Achira Lin, Siyuan Hou, Wenyi Yu, Xinnian Zhao, Haoyu Niu, Wang Geng, Longshuai Xiao, Shihai Xiao, Mangsuo Zhao, Chao Zhang
Abstract
Modern voice assistants may be shared by multiple users and should be able to answer questions about earlier conversations such as "When did I originally plan to leave?" or adapt their behavior to individual users based on past interactions. This requires more than retrieving a topically similar passage: the assistant must identify the current speaker, recover the relevant past state, and distinguish it from later revisions. We present PERSIST, a persistent memory system for multi-session, multi-speaker spoken dialogue that explicitly models Who, What, and When. PERSIST structures cross-session histories into readable event records and retrieves them with a 3W joint scoring mechanism that combines semantic content, acoustic speaker identity, and temporal state. For real-time full-duplex interaction, PERSIST further reuses intermediate representations from the dialogue backbone, avoiding query-audio re-encoding and reducing retrieval latency from 578.42 ms to 7.03 ms. We also introduce SpokenTrace, a diagnostic benchmark that factorizes evaluation along memory tasks and speaker-query types, exposing failures in recall, speaker attribution, and temporal-state tracking. On SpokenTrace, PERSIST achieves 85.08% end-to-end task accuracy and improves all-support EM@3 from 49.01% with BGE-large to 82.10%.