Text to sql struggles with ambiguous user questions in real databases

ARCS: Towards Precise Text-to-SQL via Structured Disambiguation

Computation and Language

Summary

When people ask computers to write database queries using natural language, their questions can be unclear or ambiguous, leading to wrong answers. The authors created a new way to resolve this ambiguity by using structured, specific interactions instead of open conversations. They made a special dataset called ARCS that contains real ambiguous questions and all possible correct interpretations. Their tests show that even the best models have a hard time dealing with ambiguity in text-to-SQL tasks.

What this means in practice

  • For software engineers: Develop text-to-SQL tools that handle ambiguous questions more precisely using structured clarification instead of free-form dialogue.
  • For business data analysts: Improve accuracy of querying complex databases in natural language by identifying and resolving ambiguous commands systematically.

Authors

Yihao Hu, Yanlin Feng, Naoki Otani, Nikita Bhutani

Abstract

As text-to-SQL systems move beyond demonstrations toward real-world deployment, ambiguity in user questions becomes a primary source of errors. Such ambiguities are often subtle, domain- or data-specific, and can silently cause system outputs to deviate from the user's true intent. Ambiguity is traditionally addressed through conversational clarification, which is often inefficient, cognitively demanding, and poorly aligned with real-world user workflows. We propose structured disambiguation, a new paradigm in which ambiguity is resolved through explicit, constrained interactions rather than free-form dialogue. We construct ARCS (Ambiguity Resolution Corpus for SQL), the first text-to-SQL benchmark featuring naturally occurring, unconstrained ambiguities over real-world databases, with complete annotations of all valid ambiguity points, interpretations, and SQL queries. Experimental results show that text-to-SQL remains challenging in the presence of ambiguity: gpt-6-sol achieves only 51% end-to-end execution accuracy, and no open-source model exceeds 27%.