Text to SQL evaluation improved with smarter query changes and metrics
SQLMorph: Query Mutation and Fine-Grained Metrics for Text-to-SQL Evaluation
DatabasesArtificial Intelligence
Summary
Text-to-SQL systems help people turn everyday questions into database commands, but testing how well these systems work can be tricky. The authors created SQLMorph, a method that changes questions by adding joins or tweaking language to see how systems handle these challenges. They also designed new ways to measure how accurate the systems are, beyond simply right or wrong, by looking at partial successes and mistakes. This helps developers understand where systems struggle and improve them for real-world use.
Text-to-SQLSQLquery mutationjoinnatural language processingevaluation metricsexecution accuracyprecisionrecalllarge language models
Authors
Mohammadhossein Malekpour, Mohamed Riahi, Maxime Lamothe, Amine Mhedhbi
Abstract
Text-to-SQL systems translate natural language queries into executable SQL, democratizing access to structured data. Despite recent advances driven by large language models (LLMs), evaluation remains a major bottleneck: public benchmarks fail to capture the complexity of enterprise schema, while building private evaluation sets is costly and nondeterministic, making evaluation results difficult to reproduce. To address this issue, we present SQLMorph, a framework for Text-to-SQL evaluation via query mutation. SQLMorph introduces two techniques to automatically generate and expand evaluation sets: Join Query Expansion (JQE), which systematically increases structural complexity through valid join additions, and Textual Query Augmentation (TQA), which generates controlled natural language perturbations to assess robustness to linguistic variation. JQE and TQA create targeted choke points to challenge specific system components. When applied to state-of-the-art systems, JQE increases query coverage and reveals accuracy degradation as the number of joins grows. Meanwhile, TQA shows that linguistic brittleness induced by heavy abbreviation can reduce accuracy by up to 17%. Beyond evaluation sets, SQLMorph introduces a family of execution-level metrics that address the limitations of current binary measures, such as Execution Accuracy. We define Execution Precision (EXP) and Execution Recall (EXR) to quantify the fraction of correct and recovered results, respectively, and combine them via F1 for unified scoring. Our experiments show that these relaxed metrics enable fine-grained analysis of over- and under-prediction, revealing differences across systems that binary metrics obscure. Together, SQLMorph's query mutation and fine-grained metrics support debugging and better align Text-to-SQL evaluation practices with real-world deployments.