Large language models tested on writing datalog from natural language questions
DatalogBench: Evaluating Large Language Models on Text-to-Datalog Synthesis
Programming LanguagesArtificial Intelligence
Summary
Programs written in Datalog help computers reason about tasks like analyzing other programs, but writing these programs can be difficult. The authors created DatalogBench, a test set of 136 challenges to see how well large language models (LLMs) can write Datalog programs based on natural language questions. They found that the best models get about 68% exactly right, but many errors come from missing or inconsistent pieces in the program. Better-performing models reduced such errors but still struggle with recursive reasoning tasks. DatalogBench highlights which parts of this problem remain hard for current AI tools.
What this means in practice
- •For software developers: Assist developers in automatically generating Datalog programs from natural language questions to speed up reasoning and analysis tasks.
- •For database engineers: Improve tools that translate user queries into Datalog programs for efficient data processing and logic-based querying.
Authors
Yuan Li, Hanyun Jiang, Guowei Tian, Chengpeng Wang, Peisen Yao
Abstract
Datalog underpins reasoning tasks such as program analysis, but its programs are hard to write. Existing synthesizers automate this task but require users to state their intent as input-output examples. Large language models (LLMs) suggest a more natural route, text-to-Datalog synthesis from a natural-language question, yet how well they do so has not been systematically evaluated. We present DatalogBench, a benchmark of 136 text-to-Datalog synthesis tasks curated from existing Datalog-based artifacts. Synthesized programs are graded by execution on held-out inputs against an oracle validated by mutation analysis. Across six LLMs and four prompting configurations, exact match peaks at 68.4%, and relation descriptions or an input-output example have only modest, model-dependent effects. Under direct prompting, most failures occur at compile time, typically because a model invents auxiliary predicates that it never declares or types consistently. Two coding agents reach up to 83.8% and eliminate nearly all such failures, leaving mostly semantic errors concentrated in recursive tasks. DatalogBench thus identifies recursive reasoning and decomposition as open challenges for current LLMs and agents, and offers a reliable, execution-grounded measure of both.