LLM agents struggle with changing user intentions in conversations
When Users Change Their Minds: Measuring and Repairing Intent Drift in LLM Agents
Computation and LanguageArtificial Intelligence
Summary
Sometimes when people talk with AI tools, they change their minds partway through. This causes the AI to get confused and make mistakes based on old information. The authors created a way to measure how much this happens and built a tool to keep track of what the user really wants at each step. Their tool improved AI accuracy but didn’t completely fix the problem, showing it’s a tricky challenge.
What this means in practice
- •For conversation ai developers: Improve dialogue systems by detecting and correcting intent changes during multi-turn interactions to reduce errors in responding to user requests.
- •For virtual assistant designers: Build assistants that maintain up-to-date user requirements internally to better handle evolving conversations and adjust actions accordingly.
Authors
Yanjie Zhang, Bowen Cao, Zixin Chen, Yushi Sun
Abstract
LLM agents often operate over multi-turn interactions in which user intent changes before execution. We study intent drift: the failure mode in which superseded parts of the user's intent continue to influence the final answer or tool action. We introduce IntentFlux, an executable benchmark that converts verifiable tasks into dialogues with controlled intent changes while preserving their original graders. In a 627-case calibration, mean task score falls from 0.476 to 0.384 as dialogues contain more superseded and withdrawn information. Across eight models, the rate of fully correct solutions is significantly lower when the same final task must be recovered from an evolving dialogue rather than given directly in a single turn. We further introduce StateForge, which explicitly maintains the active requirements before generation. On General-Test, it improves mean task score from 0.367 to 0.467. Providing the ground-truth final state improves performance further but still does not recover single-turn performance, indicating that state-estimation errors explain only part of the gap. These results establish intent drift as a measurable multi-turn failure mode and explicit state maintenance as a partial mitigation.