Llm agents reduce user questions while improving quality and reward
Ask Less, Ask Better: Allocating User Attention in Proactive LLM Agents with Interruption Budgets
Computation and Language
Summary
LLM agents often ask users questions to complete their tasks, but too many questions can be annoying and time-consuming. The authors introduce a way to measure the effort users spend on answering questions, distinguishing between just the number of questions and how helpful they actually are. They show that limiting the number of questions and making sure each question is relevant can save user effort with little loss in task success. Their findings suggest it’s important for computer programs to both ask fewer questions and ask better ones.
What this means in practice
- •For customer service teams: Enhance chatbot efficiency by reducing unnecessary questions while maintaining response quality in automated support scenarios.
- •For digital assistant developers: Design assistants that manage user interruptions better by limiting question frequency and improving question relevance to save user time.
Authors
Xuanming Zhang, Amir H. Rezaeian, Miguel Ballesteros, Weiyi Sun, Werner Geyer, Lydia B. Chilton, Zhou Yu, Yassine Benajiba, Dan Roth
Abstract
LLM agents can ask clarifying questions to complete tasks, but answering each question demands users' time and effort. Existing evaluations typically measure this burden by counting questions, treating necessary and avoidable questions equally. We propose an evaluation protocol that distinguishes how often agents ask from how well they ask. An interruption budget limits the number of questions, while a user-attention cost charges extra for questions that are hard to read or not worth asking, as rated by a human-validated LLM judge. Beyond question counts, this cost captures perceived effort: on conversation pairs with the same number of questions, human annotators agreed with the cost on which conversation required more effort 75% of the time. Across eight LLMs and four customer-service domains, limiting agents to one to three questions reduces user-attention cost by 36-73% while keeping 86-94% of the task reward they earn with unlimited questions. At the same budget, reviewing each question against the memory of users' earlier answers cuts attention cost by another 13% with little reward loss and yields more valuable questions. Models also differ sharply in how well they ask: at the same user-attention cost, the most efficient models gain more than twice as much reward from their questions as the others. These results show that asking less and asking better are distinct goals, and that evaluations of proactive agents should measure both.