Salus detects errors in natural language to SQL benchmarks automatically
SALUS: Automated Auditing of NL-to-SQL Benchmarks through Weak Supervision of Multi-Agent Output
DatabasesComputation and LanguageMachine Learning
Summary
Benchmarks that test converting everyday language into database queries have many mistakes, which confuse how we judge AI performance. The authors created Salus, a system that uses many AI agents to check these benchmarks and find errors without needing new human labels. Salus combines different AI votes to decide which benchmark answers are wrong, making evaluation more accurate. When tested, Salus found a large portion of errors in two popular benchmarks.
What this means in practice
- •For data platform engineers: Improve quality assurance of NL-to-SQL benchmarks to ensure reliable model evaluation and database query accuracy.
- •For database query tool developers: Automatically detect and correct benchmark annotation errors to build better natural language interfaces for databases.
Authors
Shiyuan Zhou, Ashwin Gerard Colaco, Sainyam Galhotra, Sharad Mehrotra
Abstract
Natural language to SQL (NL-to-SQL) benchmarks are foundational to progress in data analysis research, yet recent work has shown that widely-used benchmarks contain significant annotation errors. These errors silently corrupt evaluation metrics, penalize correct model output, and distort the field's understanding of state-of-the-art performance. We present SALUS, a system that automatically detects annotation errors in NL-to-SQL benchmarks. SALUS frames benchmark auditing as a weakly supervised error detection: SQL generated by multiple LLM agents drive a suite of complementary weak-labeling functions. By passing this noisy vote matrix through a generative label model, we extract high-confidence training samples without requiring human ground truth. These samples train a decision plane that maps gold SQL query features to per-agent trustworthiness, allowing SALUS to intelligently fuse reliability estimates with raw verdicts for rigorous benchmark error detection. We evaluate on BIRD-Clean-xs, a benchmark of 298 BIRD development tasks with manually verified correctness labels. SALUS achieves F1 = 0.9194, significantly outperforming the state-of-the-art baselines. Applying SALUS to the full development sets, we estimate annotation error rates of approximately 37% on BIRD and 27% on Spider.