Benchmark for question answering over databases covering reasoning and prediction
RDL-Bench: A Benchmark for Task-Aware Question Answering over Relational Databases
Machine Learning
Summary
Relational databases store organized information used in many areas, but asking them questions naturally in plain language is hard. Existing tools either handle simple lookups or specific reasoning tasks but don't combine both. The authors created RDL-Bench, a new test set where questions need either reasoning about the current data or predictions about future events. This helps evaluate systems that answer questions from databases in more realistic and varied ways.
What this means in practice
- •For enterprise software developers: Test and improve database question-answering interfaces that handle both reasoning over current data and predicting future outcomes.
- •For business analytics teams: Evaluate analytic tools that answer complex questions involving reasoning and forecasting from corporate relational databases.
Authors
Yao Cheng, Siqiang Luo
Abstract
Relational databases (RDBs) remain the primary infrastructure for storing and managing structured data across enterprise and scientific domains. Natural-language (NL) interfaces to RDBs aim to let users ask questions without writing SQL, but real user needs go beyond SQL-style lookup: some questions require reasoning over the observable database state, while others require predictive inference from historical relational records under temporal constraints. Existing benchmarks address these goals in isolation. To address this gap, we introduce RDL-Bench, a benchmark for task-aware question answering (QA) over real RDBs. RDL-Bench organizes diverse user questions into reasoning-oriented questions, whose answers are determined by the current database state, and prediction-oriented questions, whose answers depend on future outcomes under temporal constraints. Each instance contains an NL question, task metadata, schema information, temporal context, either executable evidence or target-construction logic, and denotation-level ground truth. RDL-Bench supports two evaluation settings: native solvers under their intended capabilities, and task-aware modular systems where a router dispatches each question to an appropriate execution module.