Auto researchers struggle to match human research choices in ai studies

Can Frontier Auto Researchers Reconstruct Human Research Decisions in AI?

Computation and Language

Summary

Doing scientific research often involves many decisions that humans make, like choosing what problems to study or how to solve them. The paper looks at AI systems called Auto Researchers that try to do research tasks automatically. The authors check how well these AI systems copy the decisions and work that human experts do in AI research. They find that current Auto Researchers often do not make the same choices or produce similar results as humans across different parts of the research process.

What this means in practice

  • For ai development teams: Improve tools that assist researchers by identifying where auto researcher outputs diverge from human choices in AI projects.
  • For academic conference organizers: Assess the reliability of automated research tools when vetting submissions or reviewing research workflows.

Authors

Shengyun Si, Junjie Chu, Ye Leng, Yang Zhang

Abstract

Scientific research has long been driven by substantial human effort, from reviewing prior work and identifying research gaps to designing specific solutions. Recent advances in harness-based agent systems have given rise to increasingly capable Auto Researchers, which can autonomously carry out these tasks and, increasingly, entire end-to-end research workflows. Yet despite their rapidly growing capabilities, how closely the research decisions and outputs produced by auto researchers align with those of human experts remains underexplored. We investigate this gap through the lens of research reconstruction and study: Given comparable research context, how similar are the research decisions and outputs produced by Auto Researchers to those of human researchers in AI research? To answer this question, we introduce AutoResearcher-Human Research Similarity (AutoHRS), a stage-wise evaluation framework for assessing how faithfully auto researchers reconstruct human research artifacts across different stages of the scientific workflow. Using 152 ICLR oral papers as our corpus, we generate 456 research trajectories spanning different stages of the research process. Applying AutoHRS to these trajectories, we find that current Auto Researchers frequently fail to recover key research decisions and their resulting artifacts across multiple stages of the research process.