Benchmark measures gaps in research method descriptions for coding
IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications
Computation and Language
Summary
Sometimes research papers describe new methods in a way that makes it hard for others to recreate them exactly. The authors looked at how clearly research ideas are explained so that a skilled coder can build the method without guessing. They created a test called IdeaAMBIG with many examples of unclear instructions found in real projects and made-up ones to study how well computer models can spot and fix these gaps. Their results show that while models struggle to identify missing details, they do better when asked to fix known problems. This work highlights how unclear research instructions can stop ideas from being properly implemented.
research ideamethod specificationimplementationcodification readinessreproducibilitydefect localizationclarification generationlarge language modelsbenchmarkcoding agent
Authors
Yiling Ma, Yilun Zhao, Sihong Wu, Manasi Patwardhan, Arman Cohan
Abstract
A research idea may be novel, coherent, and scientifically plausible, yet its proposed method may remain insufficiently specified for faithful implementation. We study the codification readiness of implementation-facing research-method specifications, defined by whether they provide sufficient methodological information for a competent implementer or coding agent to construct the intended method without unsupported assumptions. We construct evidence-grounded specifications and their supported resolutions from papers, codebases, issue threads, and reproduction artifacts. We introduce IdeaAMBIG, a benchmark of 660 evidence-grounded instances: 163 real-world gaps from reproducibility reports and GitHub issues, and 497 controlled synthetic gaps injected into codification-ready references. IdeaAMBIG evaluates three capabilities: codification-readiness assessment, defect localization, and clarification action generation. Defect localization receives only the specification, whereas clarification additionally receives the annotated defect. Across 13 LLMs, the best model achieves 9.6% Macro Defect Recovery Rate on real-world instances but 80.6% Macro Clarification Action Success Rate when given the defect. In an oracle study, supplying the gold resolution raises the downstream codification-ready rate from 14% to 98%. Across all evaluated models, defect localization is the main bottleneck, with stronger clarification given the defect.