Beyond the Harness: End-to-End Optimization of Context Artifacts for Enterprise Text-to-SQL
2026-08-24 • Artificial Intelligence
Artificial IntelligenceMachine Learning
AI summaryⓘ
The authors studied how large language models (LLMs) handle converting natural language queries into SQL for big business databases, which often have thousands of tables. They found that improving the context given to the model—by using past query information to create helpful summaries called SQL reference cards—helps the model perform better than just feeding it raw data. Their method worked well on real-world data from a large retailer, while results on a public benchmark were less clear. Overall, they suggest focusing on building smarter knowledge-base context from historical use rather than only tuning the model itself.
Large Language ModelsText-to-SQLKnowledge-base contextQuery-DAG decompositionSQL reference cardsQuery retrievalEnterprise benchmarksBEAVER benchmarkAST similarityDistillation procedure
Authors
Kate Gwimm, Carson Eisenach
Abstract
Deploying LLMs for enterprise Text-to-SQL is bottlenecked less by the model than by what context reaches it: business logic spans thousands of tables, and no model can ingest a full catalog at once. We argue that the most effective place to intervene is therefore the \emph{knowledge-base context} the model consumes, and that this context should be \emph{constructed} from historical usage rather than tuned for as a fixed input. Using a query-DAG decomposition--the same family of intermediates that enterprise benchmarks like BEAVER annotate, here recovered from production SQL--we compare the value of oracle query graphs versus retrieved knowledge-base context. In this ablation, retrieved knowledge-base context provides the largest marginal improvement when added to the full oracle graph. Building on this, we optimize a distillation procedure that turns historical query profiles into reusable SQL reference cards. On a benchmark of 5176 production queries from a major online retailer, optimizing these context artifacts yields larger gains (${\sim}12$--$25\%$ AST similarity) than optimizing the retrieval harness (${\sim}3$--$12\%$). On the public BEAVER benchmark, which lacks the production-usage signals available in our internal setting, the picture is more mixed: table cards alone perform about the same as raw historical SQL. The best optimized variant retrieves both cards and raw SQL, scoring $9.00\%$ versus $6.33\%$ (p-value $0.12$) for the comparable baseline on a held-out $N{=}300$ subset, using retrieved context and harness changes but no agentic loop.