Speech large language models struggle to use memory accurately in conversations
The Right Source Is Not Enough: Understanding History Interference in Speech-LLM Memory
Computation and LanguageArtificial Intelligence
Summary
Speech large language models remember what was said in earlier parts of a conversation to help answer questions. The authors found that even when these models remember an event, they sometimes use unrelated past information and give wrong answers. They showed that errors happen because memory signals affect decisions in more ways than just identifying the source. Fixing these errors requires not only remembering but also controlling how past information influences the answer.
What this means in practice
- •For voice assistant developers: Improve conversational agents by controlling how past audio information influences current responses to reduce answer errors.$Commercial implications: Enables more reliable and accurate voice assistants by refining memory use during conversation.
- •For chatbot engineers: Design chatbots that better manage which parts of conversation history affect answer choices, improving user interaction accuracy.
Authors
HongJin Song, Weiqiao Shan, Runwu Shi, Yifei Wu, Yujin Wang, Jiale Luo, Jiasheng Kuang, Xinyu Yang, Zhaoqi Cui, Chuizheng Kong, Ziyu Wu, Guowu Tan, Qiuyu Fang, Chunxiang Jin
Abstract
Speech large language models retain audio history to support multi-turn conversations. Yet remembering an event does not ensure that its information is used only when relevant. We investigate why irrelevant historical audio changes answers even when the queried event remains represented. Controlled experiments on MOSS-Audio and MiMo-Audio show that historical values direct errors toward specific competing answers, beyond a generic effect of context length. Causal interventions distinguish two paths. In open-ended question answering, most history-value errors disappear when subsequent computation cannot directly access the irrelevant source. In multiple-choice answering, many errors persist: early historical influence is carried by question-candidate states even after direct source access is blocked. Low-dimensional interventions further separate source-role information from answer-value information. Their effects peak at different depths, and changing the source-role signal does not eliminate the causal influence of competing values. On naturally occurring errors, strengthening the queried role and counteracting the historical value each improve answers; their joint intervention gives the greatest recovery. These findings support a failure of downstream evidence use beyond source identification alone: source-role information can influence a decision without exclusively controlling which historical values affect it. Reliable conversational memory therefore requires not only retention and retrieval, but also control over how retained information enters answer computation.