Papers for

business intelligence vendors

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Text to sql evaluation improved with query changes and new metrics

SQLMorph: Query Mutation and Fine-Grained Metrics for Text-to-SQL Evaluation

Abstract: Text-to-SQL systems translate natural language queries into executable SQL, democratizing access to structured data. Despite recent advances driven by large language models (LLMs), evaluation remains a major bottleneck: public benchmarks fail to capture the complexity of enterprise schema, while building private evaluation sets is costly and nondeterministic, making evaluation results difficult to reproduce. To address this issue, we present SQLMorph, a framework for Text-to-SQL evaluation via query mutation. SQLMorph introduces two techniques to automatically generate and expand evaluation sets: Join Query Expansion (JQE), which systematically increases structural complexity through valid join additions, and Textual Query Augmentation (TQA), which generates controlled natural language perturbations to assess robustness to linguistic variation. JQE and TQA create targeted choke points to challenge specific system components. When applied to state-of-the-art systems, JQE increases query coverage and reveals accuracy degradation as the number of joins grows. Meanwhile, TQA shows that linguistic brittleness induced by heavy abbreviation can reduce accuracy by up to 17%. Beyond evaluation sets, SQLMorph introduces a family of execution-level metrics that address the limitations of current binary measures, such as Execution Accuracy. We define Execution Precision (EXP) and Execution Recall (EXR) to quantify the fraction of correct and recovered results, respectively, and combine them via F1 for unified scoring. Our experiments show that these relaxed metrics enable fine-grained analysis of over- and under-prediction, revealing differences across systems that binary metrics obscure. Together, SQLMorph's query mutation and fine-grained metrics support debugging and better align Text-to-SQL evaluation practices with real-world deployments.

Tue 8 SeptDatabasesArtificial Intelligence
The gist
Text-to-SQL systems help people ask computers questions about databases using everyday language. The authors found existing methods to test these systems aren’t thorough enough, especially when queries become complex or phrased differently. They created SQLMorph, which changes queries in smart ways to see how well systems handle tricky joins and varied language. They also introduced new scoring methods to better understand when parts of answers are right or wrong. This helps developers find weaknesses and improve systems for real-world use.
Open → 2609.08950v1