Benchmark Radar indexes ai benchmark datasets scores and trends
Benchmark Radar: A Living Database and Search Engine for AI Benchmarks and Evaluation
Artificial IntelligenceInformation Retrieval
Summary
Finding the right test to evaluate artificial intelligence (AI) systems is hard because there are so many and they change quickly. The authors created Benchmark Radar, a tool that collects and organizes hundreds of AI test results and data sets in one searchable place. It updates daily from many sources and helps users understand how AI models perform on different tasks over time. The tool includes a website, daily updates, and a command-line interface for easy access.
What this means in practice
- •For ai system developers: Find and compare benchmark datasets and reported scores to design or improve AI model evaluations efficiently.
- •For machine learning engineers: Use Benchmark Radar’s catalog and score histories to select suitable benchmarks and track evaluation trends for AI deployment decisions.
Authors
Koutian Wu, Junjie Zhou, Ergan Shang, Jiayu Wang, Pengqian Han, Junkai Wang, Wanghan Xu
Abstract
Benchmark researchers and developers of large language models (LLMs) and other AI systems need to find relevant evaluations, locate their benchmark datasets and code, and understand the settings behind reported scores. We present Benchmark Radar, a living database and search engine for retrieval and discovery of AI benchmarks, covering LLM evaluation, agentic and tool-use benchmarks, coding, reasoning, safety, and domain-specific evaluations. The system combines daily discovery of benchmark papers, repositories, datasets, and releases with a searchable benchmark catalog, mentions in model cards and technical reports, and score histories. It retains source identities and citations so readers can inspect candidate benchmarks and their evaluation evidence. Daily discovery draws on 37 sources: 13 direct connectors and 24 first-party research and engineering feeds. The catalog contains 1,283 source records drawn from 4 benchmark catalogs and 12,916 numeric observations on 790 records. We describe collection and retrieval, audit the full catalog, and examine benchmark saturation, adoption trends, and the limits of score comparisons. A worked example walks through a complete prior-art search, showing how to query the catalog and inspect benchmark evidence when designing a new evaluation. We release the web dashboard with a benchmark leaderboard, a Pareto frontier view of score against measured use, saturation and trend views, daily feeds, downloadable evidence, a command-line interface (CLI) for offline queries, and reproducible analysis.