LASSO improves SQL query testing with faster data generation
LASSO: An LLM-Agent Synthesized System for SQL Validation
DatabasesSoftware Engineering
Summary
SQL queries generated by language models can have mistakes that are hard to find with only a few test datasets. The authors created LASSO, a tool that uses quick, rule-based data generation methods instead of slow constraint solvers to find these bugs more efficiently. LASSO runs various targeted tests to detect different kinds of errors and was built using careful AI-assisted coding. Their tests show that LASSO catches more problems than earlier similar tools.
What this means in practice
- •For database developers: Improve validation of automated SQL query generation by detecting subtle semantic errors with faster test data generation.
- •For quality assurance teams: Enhance robustness of SQL-based systems by applying diverse heuristic-driven test cases that reveal hidden query bugs.
Authors
Shreyas Jadhav, S. Sudarshan
Abstract
The rise of Large Language Models (LLMs) has significantly advanced text-to-SQL capabilities, but generated queries may have errors. Test datasets are important for catching errors, but having only one or a few test datasets leaves many bugs undetected. Effective testing requires multiple test datasets that can expose subtle semantic bugs and edge cases. Recent work on test data generation and equivalence checking has focused on using constraint solvers. However, constraint-based approaches are slow and not scalable. In this paper, we introduce LASSO, a framework inspired by XData, but designed to minimize the use of constraint solvers, instead prioritizing use of data generation strategies based on fast heuristics. LASSO deploys a number of "attack strategies", each targeting a specific class of semantic errors. LASSO code was built using LLM-based agentic coding. We used carefully designed workflows and rules to support effective code generation and to avoid common code generation issues. A large number of internal test cases are included in LASSO to catch errors in generated code, and more are added when we extend LASSO, to avoid subsequent regressions. Our empirical evaluation across multiple benchmarks demonstrates that LASSO outperforms the state of the art, including XData, SpotIT, ParSEval, and Polygon.