Text to sql evaluation improved with query changes and new metrics
SQLMorph: Query Mutation and Fine-Grained Metrics for Text-to-SQL Evaluation
DatabasesArtificial Intelligence
Summary
Text-to-SQL systems help people ask computers questions about databases using everyday language. The authors found existing methods to test these systems aren’t thorough enough, especially when queries become complex or phrased differently. They created SQLMorph, which changes queries in smart ways to see how well systems handle tricky joins and varied language. They also introduced new scoring methods to better understand when parts of answers are right or wrong. This helps developers find weaknesses and improve systems for real-world use.
What this means in practice
- •For enterprise software teams: Evaluate and debug text-to-SQL systems handling complex database schemas using query mutations to mimic real enterprise queries.
- •For database system developers: Measure text-to-SQL system robustness to different natural language inputs using controlled variations and detailed execution metrics.
- •For business intelligence vendors: Develop more reliable natural language interfaces for querying business data by improving evaluation with SQLMorph’s fine-grained tests and scores.$Commercial implications: This paper enables improved natural language query tools that vendors can sell to enterprises for better data access.
Authors
Mohammadhossein Malekpour, Mohamed Riahi, Maxime Lamothe, Amine Mhedhbi
Abstract
Text-to-SQL systems translate natural language queries into executable SQL, democratizing access to structured data. Despite recent advances driven by large language models (LLMs), evaluation remains a major bottleneck: public benchmarks fail to capture the complexity of enterprise schema, while building private evaluation sets is costly and nondeterministic, making evaluation results difficult to reproduce. To address this issue, we present SQLMorph, a framework for Text-to-SQL evaluation via query mutation. SQLMorph introduces two techniques to automatically generate and expand evaluation sets: Join Query Expansion (JQE), which systematically increases structural complexity through valid join additions, and Textual Query Augmentation (TQA), which generates controlled natural language perturbations to assess robustness to linguistic variation. JQE and TQA create targeted choke points to challenge specific system components. When applied to state-of-the-art systems, JQE increases query coverage and reveals accuracy degradation as the number of joins grows. Meanwhile, TQA shows that linguistic brittleness induced by heavy abbreviation can reduce accuracy by up to 17%. Beyond evaluation sets, SQLMorph introduces a family of execution-level metrics that address the limitations of current binary measures, such as Execution Accuracy. We define Execution Precision (EXP) and Execution Recall (EXR) to quantify the fraction of correct and recovered results, respectively, and combine them via F1 for unified scoring. Our experiments show that these relaxed metrics enable fine-grained analysis of over- and under-prediction, revealing differences across systems that binary metrics obscure. Together, SQLMorph's query mutation and fine-grained metrics support debugging and better align Text-to-SQL evaluation practices with real-world deployments.