LLM-generated SQL queries repaired with precise property checks and fewer errors
Post-SQLFix: Property-Guided Repair of LLM-Generated SQL with Runtime Verification
Software Engineering
Summary
Generating SQL queries with AI models often produces mistakes that are hard to fix because error messages can be unclear or too specific to a SQL type. The authors introduce a new way to automatically fix these errors by checking the queries against a set of clear rules and properties before accepting any changes. This method helps reduce errors more effectively and uses fewer attempts to correct queries. As a result, the approach improves the accuracy and reliability of AI-generated SQL queries.
What this means in practice
- •For database developers: Fix SQL queries generated by AI with fewer repair attempts and more precise error checking across SQL dialects.
- •For customer support engineers: Automatically correct user-submitted SQL queries in support tools by verifying fixes against detailed structural and semantic rules.
Authors
Zhihao Guo, Zonghan Wu, Yilei Shao, Haizhou Du, Nan Tang, Xiao Cheng, Huan Huo
Abstract
Large Language Models (LLMs) for Text-to-SQL frequently generate queries with syntax errors, invalid identifier references, or violations of SQL scoping and typing rules. Repairing these errors is a program correctness problem, yet existing approaches rely on Execution Feedback (EF), whose symptom-level, dialect-specific error messages force the LLM to speculate on root causes. We identify an Arbitration Paradox in this paradigm: the system relies on unverified models both to propose repairs and to make the final judgment on whether those repairs satisfy the required properties. We resolve this paradox by reformulating Text-to-SQL repair as property-guided program repair. Our framework, Post-SQLFix, canonicalizes SQL into a dialect-aware Canonical Query Structure (CQS) and validates it through a hierarchical static analyzer with three stratified property layers (Syntactic, Context-Free Structural, Context-Sensitive Semantic). The analyzer produces precise, schema-aware violation diagnostics to guide the LLM's repair proposals. A Verify-Before-Apply Temporal Monitor then accepts only repairs that reduce the violation count, do not reintroduce resolved violations, and satisfy the operator contracts. These runtime invariants are bounded by the declared rule family and operator contracts, while a finite retry budget ensures halting. On three established benchmarks (BIRD, Spider, Mini-Dev) spanning over 3,000 evaluation instances, three SQL dialects (SQLite, MySQL, PostgreSQL), and six model configurations, Post-SQLFix resolves 77.8% of the execution errors that EF leaves unrepaired, while requiring less than half as many repair iterations and 30% fewer tokens. Crucially, accepted transitions under the monitor introduce no statically detectable structural regressions in our evaluation, translating checked repair fidelity into absolute accuracy improvements of up to +11.6 percentage points.