Benchmark tests multi database agents for finding and using data correctly

DSABench: Benchmarking Source Discovery, Coordination, and Execution for Multi-Database Agents

Artificial Intelligence

Summary

When computers answer questions by searching databases, they often assume they already know which databases to use. This paper shows that in real situations with many databases, the computer first needs to figure out which databases to use, how to combine their answers, and then run the full process correctly. The authors created DSABench, a new benchmark with complex tasks requiring multiple databases, to test how well computer agents handle this challenge. Their tests reveal big gaps between just finding databases and correctly combining and executing data from them, and that asking clarifying questions helps improve results on tricky tasks.

What this means in practice

  • For enterprise data engineers: Evaluate multi-database query agents to improve data source discovery and query coordination across large organizational catalogs.
  • For cloud platform teams: Test and improve multi-source query execution for services that integrate data from many cloud databases to answer user questions.$Commercial implications: Enables building commercial cloud data integration tools that automate multi-database querying with minimal user input.

Authors

Jinlong Luo, Haoran Wu, Bo Xu

Abstract

Most text-to-SQL benchmarks provide the target database. In a large data catalog, however, choosing the sources is part of the problem: an agent must find every database needed for a request, determine how their results fit together, and execute the resulting workflow. We present DSABench to evaluate these capabilities jointly. Its 815 human-reviewed tasks draw on Spider, BIRD, and LiveSQLBench; 618 require several databases to contribute, and 105 allow more than one interpretation of the required sources. Executable contracts define source participation, cross-source dependencies, and source-local execution, while RouteEnv lets agents inspect public catalog evidence without revealing the gold annotations. In our routing-only study across seven model backends, the highest RA-Set score is 60.55%, compared with a highest RA-Graph score of 34.66% when coordination requirements are included. The gap becomes more pronounced when execution is required. Across four backends, median non-privileged Route+Exec-Graph success ranges from 6.69% to 7.98% without clarification and from 11.44% to 13.25% with one clarification; the best observed clarified configuration reaches 16.99%. On ambiguous tasks, clarification improves exact branch-source recovery by 39.8-43.2 percentage points. These findings illustrate why retrieving plausible databases or producing correct local SQL is not enough: an agent must carry the right information through the entire multi-database workflow.