Agentic retrying drives query improvement more than feedback style
What Drives Recovery in Agentic Text-to-Cypher? LAST-CQ: An LLM Agent Self-Refinement Framework
Artificial IntelligenceComputation and LanguageMachine LearningMultiagent SystemsSoftware Engineering
Summary
Generating database queries from text can be hard because the system might make mistakes. The authors study a multi-agent system that tries queries, notices failures, and retries to fix problems. They find that simply detecting failures and retrying is what leads to better results, not complex feedback or more guesses. Their method recovers most queries that initially fail, making the system much more reliable. They also show that some common ways to measure success can be misleading.
What this means in practice
- •For database engineers: Improve text-to-query systems by focusing on failure detection and retry rather than complex feedback mechanisms.
- •For software developers: Build more reliable text-to-database interfaces by integrating execution-grounded retry loops that reduce failed queries without extra calls.
Authors
Ioannis Prokopiou, Athanasios Aidinis, Panagiotis-Christos Kyrmpatsos, Pantelis Vikatos
Abstract
Agentic pipelines for structured-query generation are rapidly expanding, but it is unclear which part of the loop produces the gain. We use LAST-CQ -- a five-agent, training-free, execution-grounded Text-to-Cypher framework -- as an instrumented testbed, running three counterfactuals over 2,471 live-database queries and six backbones spanning three vendor scale tiers. Removing correction is worth between 3.1% aggregate execution-BLEU against the single-pass system and 12.3% against a no-refinement counterfactual (up to 80.7% for the weakest backbone). Replacing schema-grounded, LLM-synthesised feedback with raw database error strings costs almost nothing (20.9% vs. 19.9% naive exact match; <0.2% end-to-end; equivalent within $\pm 0.075$ set-F1 by two one-sided tests). Spending the same call budget on parallel sampling degrades quality by 10-11%. What works is detecting failure and routing it to a retry, not the feedback sophistication or number of samples. LAST-CQ itself recovers 91.7% of queries that fail under single-pass generation, while a query that succeeds first time still costs exactly one LLM call. We also show that n-gram overlap on serialised results is not a bound in either direction: it over-scores against set equivalence on 65.9% of results while under-scoring against judged semantics. Finally, we calibrate our LLM judge against blind human labels and find it optimistic by 9 points.