Conversational AI improves asking questions to clarify table queries
Conversational Task Disambiguation over Tabular Data: Leakage-Aware Formulation, Benchmark Suite, and Training
Machine LearningArtificial IntelligenceComputation and Language
Summary
Sometimes when people ask computers questions about data tables, the questions are unclear or missing some details. The authors created a way to measure and improve how well AI agents ask follow-up questions to figure out what the user really wants before giving an answer. They combined multiple datasets into one benchmark and trained a system to ask better questions, which helped the AI understand tasks more accurately and made sure it wasn’t cheating by using extra hints. This work helps AI systems get clearer information before trying to solve tasks involving databases.
What this means in practice
- •For database developers: Improve conversational interfaces by training AI to ask clarifying questions when users query databases using natural language.
- •For customer support teams: Enable virtual agents to identify and resolve unclear customer requests about data or reports more accurately through targeted clarification dialogue.
Authors
Nafiseh Ghoroghchian, Luis Scoccola, Tina Sedaghat, Omid Vaheb, Hannah Chen, Dino D'Agostino, Keyvan Golestan
Abstract
Conversational task disambiguation over tabular data uses dialogue to resolve missing information about a user's intended task before producing a solution over tables or databases. Existing evaluation and training lack a leakage-aware foundation. Task success mixes the agent's disambiguation and solution-generation capabilities and can also reflect oracle leakage, that is, information that a user simulator reveals beyond what a real user would. Existing datasets also lack a shared representation of ambiguities and access boundaries. We introduce the notion of an ambiguous verifiable task, which formalizes ambiguities and resolutions, decomposing the agent into an asking policy and a solution policy, and the environment into an oracle and verifier. This framework provides baselines and metrics for evaluating task disambiguation separately from solution generation, formal definitions of oracle leakage, judge-free leakage diagnostics, and a training objective for the asking policy. We instantiate the framework in text-to-SQL with AmbiTab, a benchmark suite that unifies six ambiguous datasets under a common representation specifying what the agent, oracle, and verifier may access. We evaluate clarification strategies and oracle leakage, and train an asking policy with reinforcement learning. The trained asker improves our disambiguation metrics on all six datasets and task success on five, and our leakage diagnostics measure how training affects oracle leakage.