Multi-turn conversations reduce accuracy of retrieval-augmented language models
Lost in Conversation or Lost in Translation? Diagnosing Multi-Turn Degradation in RAG
Computation and LanguageArtificial Intelligence
Summary
When people ask complicated questions step-by-step in conversations, computer systems that combine search with language models often perform worse than when answering single, clear questions. The authors studied this problem by simulating many conversations and found that the systems become less accurate and reliable over multiple turns. They discovered two main issues: sometimes the way people rephrase questions confuses the search part, and sometimes the search works but the language model struggles to combine all the information correctly.
What this means in practice
- •For conversational ai developers: Improve multi-turn dialogue systems by diagnosing when retrieval queries become distorted or synthesis fails in question answering pipelines.
- •For knowledge management teams: Detect and mitigate reliability drops when deploying retrieval-augmented LLMs for interactive question answering over corporate data.
Authors
Pranav Handa, Ariful Azad
Abstract
When conversing with large language models (LLMs), users often begin with a simple question and build towards a multi-hop question through follow-up turns. Retrieval-augmented generation (RAG) and its graph-based variant (GraphRAG) have become the dominant approaches for grounding LLM responses in external evidence, yet both are evaluated almost exclusively on single-turn, fully specified queries. We systematically investigate this evaluation mismatch through a large-scale simulation study. Building on prior work on multi-turn LLM evaluation, we transform questions from multi-hop question answering (QA) benchmarks into underspecified conversations and evaluate ten LLM assistants with eight retrieval systems across 1.5 million simulated conversations. Our findings reveal that multi-turn interaction causes widespread performance degradation, incurring relative performance drops of up to 21% and increasing unreliability by 47%, making RAG systems simultaneously less accurate and less reliable. We identify two distinct failure modes behind this degradation. Systems are either lost in translation, where conversational rephrasing distorts the retrieval query, or lost in conversation, where retrieval succeeds but the LLM fails to synthesize evidence distributed across turns.