Self refining prompts improve streaming text to SQL accuracy significantly
Self-Refining Prompt Induction for Streaming Text-to-SQL
Computation and LanguageArtificial IntelligenceDatabases
Summary
Understanding and converting natural language questions into correct SQL queries for streaming data is tricky because streaming SQL has many special rules. The authors developed a method that improves the instructions given to language models by learning from their mistakes without changing the model itself. This approach increased accuracy across different models and domains using a refined prompt. They also reduced the prompt's size while keeping performance high. Their method is already being tested in a real assistant tool.
What this means in practice
- •For data engineers: Improve text-to-SQL assistants to handle streaming SQL dialects with better accuracy without retraining models.
- •For cloud platform teams: Enhance automated query generation in streaming data platforms by integrating self-refined prompts tailored to specific SQL dialects.
- •For customer service chatbot developers: Build chatbots that convert user questions into streaming SQL queries accurately by using self-refining prompt techniques.$Commercial implications: Enables creation of commercial chatbot products that generate precise streaming SQL queries from natural language, improving customer interaction capabilities.
Authors
Gaetano Rossiello, Alfio Gliozzo, Nandana Mihindukulasooriya, Erick Lee, Nabila Siregar, Lisa Amini
Abstract
Text-to-SQL assistants for streaming engines must handle dialect-specific types, windowing rules, and temporal semantics, including constructs poorly handled by generic prompts. We present Self-Refining Prompt Induction (SRPI), an offline method that converts generation failures into reusable dialect instructions while keeping model weights fixed. A Refiner revises the Generator's prompt using reference-conditioned feedback from three LLM judges; a sampled regression guard checks earlier successes, and a separate Compressor reduces prompt length. Inference uses the frozen prompt without retrieval or query repair. We evaluate cross-domain and cross-model transfer on Confluent Cloud Flink SQL using a controlled synthetic workload seeded by 30 expert-written examples, with 335 induction examples and 257 questions from held-out domains. The same induced prompt improves mean judge scores across seven Generators by 7.1-19.2 percentage points, including an increase from 85.3% to 97.5% for the induction model. A separate compression study reduces prompt length by 52.7% with a 0.19-percentage-point score decrease. An assistant using the induced prompt is in early access, with initial user feedback to inform subsequent refinement.