Evaluating LLMs on Conversational Text-to-SQL under Chain Ambiguity and Intent Drift
Computation and LanguageArtificial IntelligenceDatabases
Summary
The authors created TIDE-Bench, a new test to better evaluate how well language models handle tricky conversations when people ask databases questions that change or get unclear over multiple turns. Unlike previous tests that only checked if the final answers were correct, this benchmark also looks at how models deal with unclear questions needing extra clarifications (chain ambiguity) and when users change their mind mid-conversation (intent drift). They tested 12 language models and found these models struggle to identify these shifts consistently, even with several clarifications. The authors share TIDE-Bench’s data and tools so others can study and improve models in these areas.
Authors
Yujia Liu, Jiayan Lin, Zijin Hong, Zheng Yuan, Shengyuan Chen, Hao Chen, Qinggang Zhang, Xiao Huang, Feiran Huang
Abstract
Recent advances in large language models (LLMs) have established conversational text-to-SQL as a practical interface between users and databases, often involving multiple turns of clarification and revision. However, existing benchmarks primarily evaluate execution accuracy, leaving the unfolding and shifting of user intent across turns largely uncovered. To address this, we introduce TIDE-Bench, a benchmark for conversational text-to-SQL under chain ambiguity and intent drift evaluation, targeting two recurring patterns: chain ambiguity, where an underspecified question triggers layered clarification with conditional dependencies, and intent drift, where the user retracts and replaces a previously committed request element. Built on 514 anchor SQLs from BIRD, TIDE-Bench comprises 1,542 samples and introduces dedicated metrics for chain identification and drift recognition-resolution beyond execution accuracy. Evaluating 12 advanced LLMs reveals a persistent chain identification bottleneck unaffected by clarification frequency, a wide drift recognition-resolution gap, and overlap between failure modes when jointly activated. The corresponding code of TIDE-Bench is released for further research.