RubricAtlas improves evaluation of deep research agent reports
RubricAtlas: Mining Type-Level Rubrics from Expert Reports to Evaluate Deep Research Agents
Artificial IntelligenceComputation and Language
Summary
Evaluating complex research reports made by AI agents can be tricky because they do more than just find facts—they also analyze and explain. The authors created RubricAtlas, which learns common analysis questions from many expert reports on similar types of research. This helps check whether AI reports cover all important analytical points without guessing the right answers. Their tests show that different expert reports emphasize different analyses, and RubricAtlas can fairly compare AI research reports based on completeness of their analysis.
What this means in practice
- •For enterprise research teams: Compare and assess completeness of analytical reports generated by AI across similar types of research tasks.
- •For financial analysis teams: Use RubricAtlas to audit and improve the coverage of analytical criteria in financial research documents and AI-generated summaries.
Authors
Delin Wang, Sinuo Wang, Rui Sun, Beidi Luan, Qiming Shi, Jing Li, Daxin Jiang, Zuo Bai
Abstract
Deep research agents conduct multi-step Web searches and synthesise retrieved information into analytical reports. Evaluating these outputs requires checking which analyses they deliver, beyond the presence of relevant facts. Query-specific binary rubrics make individual requirements explicit, but task-level checklists may leave the overall analytical structure underexamined. We introduce RubricAtlas, which derives type-level analytical requirements from expert reports on different objects within the same research type and organises detailed checks around recurring analytical questions. The requirements give related tasks a shared standard for what to analyse, without prescribing their answers. Our framework links binary checkpoints to report passages and evaluates completeness separately from argument validity. Built from 3,079 Chinese financial research reports, RubricAtlas covers 99 research types. Reference-report selection experiments show that individual reports cover only about half of their type's analytical questions, and that selecting requirements from different reports changes evaluation scores and agent rankings. Across ten agents, two judge pipelines achieve checkpoint agreement of $κ$ = 0.71 and agent-ranking agreement of Kendall $τ$ = 0.87. Controlled deletions reduce scores on the affected analyses, while padding produces no detectable increase. These findings support type-level rubrics for comparing the analytical completeness of Web deep research outputs, with individual decisions traceable to report evidence.