Benchmark reveals challenges for language models querying time-series databases
TQTS-Bench: A Multi-Syntax Benchmark for Text-to-Query over Time-Series Databases
Computation and Language
Summary
Understanding questions about time-based data is hard for current language models. The authors created a big set of over 6,000 questions covering many types of time-series databases and query styles to test these models. Even the best models only answered about half correctly, while humans scored much higher. The main problems were dealing with different database query languages, understanding questions about time accurately, and linking questions to the right data structures. This work helps highlight where improvements are needed for better tools to ask questions about time-series data.
What this means in practice
- •For data engineering teams: Evaluate and improve text-to-query tools handling diverse time-series databases for accurate data retrieval.
- •For business intelligence teams: Validate natural language querying solutions for time-relevant business metrics across varied time-series data sources.
Authors
Fei Lyu, Zhiyi Peng, Jiaming Liu, Yixuan Yang, Changjian Chen, Zhuo Tang, Jiapeng Zhang, Kenli Li
Abstract
Large language models (LLMs) have significantly advanced natural language querying over relational databases, yet their ability to query time-series databases (TSDBs) remains largely unassessed. Existing benchmarks fail to adequately capture the non-unified query syntaxes, diverse application domains, and unique time-specific query intents inherent to TSDBs. To address this gap, we introduce TQTS-BENCH, a multi-syntax benchmark for evaluating text-to-query capabilities over TSDBs. TQTS-BENCH contains 6,125 high-quality question-answering (QA) pairs spanning 97 TSDBs, 23 distinct query syntaxes, 22 application domains, and 4 types of time-specific query intents. It is constructed through a human-centric AI-assisted workflow, where all QA pairs are carefully reviewed and revised by domain experts to ensure quality and correctness. Extensive evaluations of advanced LLMs and state-of-the-art text-to-query methods reveal challenges in querying TSDBs. Even the best-performing model evaluated, Claude-Opus-5, achieves only 48.98% execution accuracy, while humans reach 87.34%. Error analysis reveals that this performance gap mainly stems from the heterogeneous query syntaxes across different TSDBs, misinterpretation of time-specific intents, and incorrect schema linking. These findings highlight new opportunities to narrow the gap between current LLM capabilities and the requirements of TSDB queries in real-world applications. The benchmark is available at: https://anonymous.4open.science/r/TQTS-Bench-00CD.