HarnessSQL improves sql agent accuracy by training with live databases
HarnessSQL: Harness-Native Training for SQL Agents in Realistic Database Environments
Computation and Language
Summary
Many AI programs that turn questions into database queries learn by making a single guess, but real use involves asking many questions and trying out queries step-by-step. The authors found that this difference makes these programs less accurate when used in practice. They created HarnessSQL, a method that trains these programs by letting them practice interacting with live databases during learning. This approach greatly improved the accuracy of compact models and worked well even on new types of databases.
What this means in practice
- •For database application developers: Increase the accuracy of conversational database query tools by training AI agents with live database interactions instead of static queries.
- •For enterprise data teams: Build smarter database assistants that can handle complex workflows involving multiple question and query attempts by training agents in realistic environments.
Authors
Haolin Yang, Jipeng Zhang, Jian Xie, Shuaishuai Gong, Sirui Han, Yike Guo
Abstract
Text-to-SQL models are commonly trained to map questions directly to static queries, whereas real-world database agents operate through stateful, multi-turn interaction with live databases -- inspecting schemas, executing probe queries, diagnosing errors, and revising hypotheses. This creates a critical train-deploy mismatch, as the execution harness that mediates this interaction is introduced only at inference time. To bridge this gap, we propose HarnessSQL, a harness-native post-training framework that preserves the full interaction structure throughout both supervised fine-tuning and reinforcement learning. HarnessSQL builds isolated, executable database environments paired with hidden execution oracles, rolls out teachers directly inside the target SQL harness, and retains only verified trajectories for full-sequence SFT, followed by execution-reward RL. Across Spider 2.0-SQLite, HarnessSQL dramatically boosts the execution accuracy of compact models, raising Qwen3-8B from 15.5% to 45.2% and Qwen3-14B from 22.2% to 54.8%, while transferring effectively to out-of-distribution interactive benchmarks such as BIRD-Interact and LiveSQLBench. Our findings demonstrate that training database agents directly within their execution harness is essential for mastering complex, long-horizon database workflows.