AI summaryⓘ
The authors highlight that traditional ways of checking SQL query accuracy don’t cover many real-world problems like ambiguous questions or outdated data. They created WarehouseReliabilityBench, a set of 400 test tasks with tricky cases needing clarification or refusals to answer. Their AI agent, QueryProof, uses rules from the database schema and checks answers carefully after execution, reducing wrong or misleading answers compared to a more powerful but less structured model. The study shows that adding a deterministic checking layer helps improve reliability more than just increasing model size. However, some parts like routing didn’t significantly affect results, and the improvements rely on this rule-based approach.
LLM analytics agentsSQL syntax accuracyWarehouseReliabilityBenchQueryProof agentsemantic layerpost-execution checksBusiness Truth Rateclarification and abstentiondeterministic layerpaired bootstrap
Abstract
LLM analytics agents are evaluated on SQL syntax accuracy, but production failures look different: questions with two valid business definitions, questions the warehouse cannot answer, deprecated columns after a schema change, and queries that execute successfully while returning the wrong business number. No execution-match metric can score them. This paper introduces WarehouseReliabilityBench, 400 frozen tasks over two synthetic warehouses in which roughly half the correct responses are a clarification, an abstention or a refusal, with pinned denominators and a pre-registered paired bootstrap fixing each claim verb before the numbers existed. QueryProof, a 7B agent, uses rules derived from a semantic layer and physical catalog to determine its behaviour, and gates every answer on deterministic post-execution checks. On an 80-task synthetic test split evaluated once, QueryProof outperforms a direct-prompted 32B baseline by +0.237 [+0.112, +0.375] Business Truth Rate at 71.0% lower cost per correct answer; against a cost-matched few-shot baseline the accuracy gain holds but the cost difference does not resolve. This compares systems rather than model sizes: the 32B baseline receives none of the scaffolding. False success falls from 0.754 to 0.351 of returned answers, and no wrong number was returned on an answerable task (0 of 24), though 13 answers went to questions requiring clarification or abstention. Removing the routing layer changes little (0.562 against 0.537), so the result does not depend on escalation. Routing tuned on validation over-abstains on test, and the fitted confidence model loses to the heuristic it replaced. Resampling template families rather than tasks widens both accuracy intervals to include zero, so the effect's direction is better supported than its magnitude. The gain tracks the deterministic layer, though no component ablation was run.