Evaluation method improves ai augmented text to sql query accuracy
The Stochastic Shift: A New Evaluation Paradigm for Text-to-SQL with AI Operators
DatabasesArtificial Intelligence
Summary
Text-to-SQL systems translate natural language questions into database queries, but when these queries include AI operators that produce flexible and non-deterministic results, existing evaluation methods often fail to judge them correctly. The authors show that current accuracy metrics can wrongly reject many valid queries. They propose a new way to evaluate these AI-augmented queries by separating the traditional database logic from the AI-generated parts, leading to much more reliable assessments. This new evaluation method works well on multiple platforms, detecting correct queries with high accuracy.
What this means in practice
- •For data engineering teams: Improve validation of AI-augmented SQL queries for reliable data analytics workflows in production systems.
- •For enterprise software developers: Develop tools that accurately benchmark text-to-SQL generators incorporating AI functions across different database platforms.$Commercial implications: Enables creation of commercial benchmarks and testing tools for AI-empowered query generation products.
Authors
Tarfah Alrashed, Fatma Ozcan, Per Jacobsson, Tal Neiman, Xianshun Chen
Abstract
SQL has been augmented with AI operators, enabling modern data analytics platforms to derive insights from both structured and unstructured data. We observe that while current Text-to-SQL systems can successfully generate these AI-augmented queries, reliably evaluating their correctness remains a critical open challenge. Current metrics, which rely on exact query results and deterministic execution, systematically fail against the flexible, non-deterministic outputs of AI operators. In this paper, we formalize these unique evaluation failure modes and introduce a Multilayered Evaluation Framework that decouples deterministic database logic from flexible AI semantics. We test our approach across both industry (BigQuery) and academic (ThalamusDB) systems. We demonstrate that traditional Execution Accuracy severely penalizes valid queries, achieving as low as a 25% detection rate for correct translations. Furthermore, even a state-of-the-art LLM-based autorater falsely rejects 32% of accurate queries due to the complexity of judging both relational and AI components simultaneously. By validating the standard relational logic and the AI operations separately, our framework achieves state-of-the-art overall accuracy across both platforms (up to 97.2%), proposing a reliable standard for benchmarking AI-powered SQL generators.