Agent based system benchmarks reproducibility of scientific experiments
Overview of the NLPCC 2026 Shared Task 11: Agent-Based Experiment Reproduction from Scientific Papers
Computation and Language
Summary
Reproducing scientific experiments is important but gets harder as studies become more complex. The authors put together a new way to test AI agents that try to repeat experiments by following research papers. They created a set of 150 papers from machine learning and science fields, letting agents be tracked step-by-step as they try to reproduce findings. Their tests show current AI agents still face challenges, especially when running code, but their method for scoring agent performance matches well with human judgments.
What this means in practice
- •For machine learning engineers: Evaluate and improve AI systems that automatically reproduce machine learning experiments from scientific papers using detailed behavioral benchmarks.
- •For data science teams: Use a rubric-based framework to track and validate automated experiment reproduction processes in AI4Science projects.
Authors
Hanhua Hong, Yizhi Li, Luu Gia Huy, Jian Yang, Ming Zhou, Chenghua Lin
Abstract
Reproducibility is essential to scientific progress, yet the growing volume and complexity of scientific publications make exhaustive manual verification increasingly impractical. Although recent advances in large language model (LLM) agents enable automated experiment reproduction, existing evaluations largely focus on final repositories and are typically limited to machine learning (ML). We introduce AgentActionBench, a process-oriented benchmark for evaluating agent-based experiment reproduction across ML and AI4Science domains. Our framework uses an MCP-based Action Recorder to capture agents' behaviour throughout the reproduction process and evaluates the resulting traces with paper-specific rubrics. AgentActionBench contains 150 papers, including 120 ML papers and 30 AI4Science papers. A human-annotated subset covering 10% of the benchmark provides validation data, while model-assisted augmentation expands the full benchmark to more than 10,000 rubric items. Experimental results show that current systems remain limited, with execution as the primary bottleneck. Meanwhile, the strong Pearson and Spearman correlations between model-generated and human-annotated rubrics validate the reliability of our scalable rubric-generation approach.