From Simple QA to Deep Research: A Verifiable Benchmark Constructed through Iterative Task Evolution
2026-08-03 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors created a new set of 500 challenging research tasks across many subjects to test deep understanding in AI models. They built these tasks automatically using a special process that turns simple questions into complex, step-by-step problems with clear checkpoints. The tasks are structured like flowcharts, making it easier to track and evaluate answers reliably. Their experiments show this benchmark can tell how well different models perform and provides detailed, consistent scoring. All their data and code are made available for others to use.
benchmarkdeep research tasksevaluationdirected acyclic graph (DAG)automatic constructioniterative pipelinequery formstask-specific knowledgefact-grounded rubricsmodel discrimination
Authors
Can Wang, Haoran Chen, Haowen Gao, Hao Ding, Zhaoyang Liu, Zhiying Tu
Abstract
Deep research benchmarks require expert-level tasks and reliable evaluation grounded in task-specific knowledge. Existing benchmarks rely heavily on expert authoring or pre-existing human-authored materials, while fully automatic construction struggles to ensure consistent and traceable verification. To address this gap, we introduce a verifiable benchmark of 500 deep research tasks spanning 31 topics and 10 major categories, with three query forms designed to probe complementary capabilities required for deep research. The benchmark is constructed automatically using an iterative Explorer-Formalizer-Challenger pipeline that progressively transforms simple questions into deep research tasks. Each task is represented as a directed acyclic graph (DAG) of atomic steps and associated checkpoints, enabling the query, DAG, and rubrics to evolve together in a controlled manner. Experiments demonstrate that the benchmark clearly discriminates among models and query types, while its fact-grounded pointwise rubrics enable fine-grained, human-aligned, and stable evaluation. Our data, implementation, and results are publicly available.