Agentic text to sql systems improve question handling to reduce hidden errors
Beyond Correctness: Resolving Underspecification in Agentic Text-to-SQL
Computation and Language
Summary
Sometimes, systems that turn natural language questions into database queries don’t ask enough questions to clarify unclear requests. This can lead the system to guess answers that seem correct but may not truly match what the user wanted. The authors show that these systems often stop asking questions too soon or forget to ask important ones. They introduce a new method called PlanPool, which keeps track of questions the system plans to ask and ensures they are all addressed before giving a final answer. This approach improves how well the system covers unclear parts of a question and reduces silent errors while still answering accurately.
What this means in practice
- •For database application developers: Build interactive database query tools that better clarify ambiguous user requests before generating SQL commands.
- •For customer support automation teams: Create AI agents that ask planned clarifying questions to reduce errors in answering user database queries.
Authors
Wen-Zhi Li, Yue Gong, Konstantinos Kanellis, Balakrishnan Murali Narayanaswamy
Abstract
Agentic Text-to-SQL systems can interact with users to clarify underspecified queries before generating SQL. However, a correct execution result does not necessarily imply that the agent has adequately resolved the underlying underspecification: the agent may silently make unverified assumptions that happen to match the intended answer. We show that this behavior is driven in part by premature clarification termination. Although forcing an agent to ask more questions improves execution accuracy, ambiguities are concentrated in earlier interactions, making brute-force questioning inefficient. More importantly, even when explicitly prompted to plan its clarification process, the agent frequently abandons questions that it has already identified as relevant. To address this failure mode, we introduce PlanPool, which externalizes the clarification plan as a mutable question pool. Every planned question must be explicitly asked or dropped before submission, while newly discovered ambiguities can be added during interaction. Across three benchmarks derived from BIRD-Interact and Spider, PlanPool consistently improves ambiguity coverage and reduces silent failures over unconstrained and prompt-based alternatives, while maintaining competitive execution accuracy. Our results highlight an important distinction in agentic reasoning: identifying missing information is not sufficient, and the agent must also reliably maintain and resolve it before committing to an answer.