SpotIt+: Verification-based Text-to-SQL Evaluation with Database Constraints
2026-03-04 • Databases
DatabasesArtificial IntelligenceLogic in Computer ScienceProgramming Languages
AI summaryⓘ
The authors created SpotIt+, a free tool that checks if a computer-generated SQL query matches the correct one by looking for examples of databases where they behave differently. They improved this tool by teaching it to find realistic rules that the databases should follow, using both manual rule-finding and AI help. Tests showed their method finds more practical differences between queries that other tests might miss. This helps better understand when and why automated SQL answers are wrong.
Text-to-SQLSQL querybounded equivalence verificationconstraint miningrule-based specificationlarge language models (LLM)counterexample generationdatabase instanceBIRD datasetquery evaluation
Authors
Rocky Klopfenstein, Yang He, Andrew Tremante, Yuepeng Wang, Nina Narodytska, Haoze Wu
Abstract
We present SpotIt+, an open-source tool for evaluating Text-to-SQL systems via bounded equivalence verification. Given a generated SQL query and the ground truth, SpotIt+ actively searches for database instances that differentiate the two queries. To ensure that the generated counterexamples reflect practically relevant discrepancies, we introduce a constraint-mining pipeline that combines rule-based specification mining over example databases with LLM-based validation. Experimental results on the BIRD dataset show that the mined constraints enable SpotIt+ to generate more realistic differentiating databases, while preserving its ability to efficiently uncover numerous discrepancies between generated and gold SQL queries that are missed by standard test-based evaluation.