Implementation gaps limit coding of research methods available solutions lag
IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications
Computation and Language
Summary
Sometimes research papers describe methods that sound good but don’t give enough detail for someone to actually build the method correctly. The authors studied how often these important missing details happen and created a collection of examples showing these gaps. They tested different large language models to see if the models can spot these missing steps and help fix them. The models often struggled to find the missing parts but did better at suggesting fixes once told what’s missing. This shows that understanding what part of a method is unclear is the hardest problem.
What this means in practice
- •For software developers: Detect unclear steps in research method descriptions to guide accurate coding of scientific software implementations.
- •For quality assurance teams: Use automated tools to find and fix specification gaps that cause reproducibility issues in code linked to research papers.
Authors
Yiling Ma, Yilun Zhao, Sihong Wu, Manasi Patwardhan, Arman Cohan
Abstract
A research idea may be novel, coherent, and scientifically plausible, yet its proposed method may remain insufficiently specified for faithful implementation. We study the codification readiness of implementation-facing research-method specifications, defined by whether they provide sufficient methodological information for a competent implementer or coding agent to construct the intended method without unsupported assumptions. We construct evidence-grounded specifications and their supported resolutions from papers, codebases, issue threads, and reproduction artifacts. We introduce IdeaAMBIG, a benchmark of 660 evidence-grounded instances: 163 real-world gaps from reproducibility reports and GitHub issues, and 497 controlled synthetic gaps injected into codification-ready references. IdeaAMBIG evaluates three capabilities: codification-readiness assessment, defect localization, and clarification action generation. Defect localization receives only the specification, whereas clarification additionally receives the annotated defect. Across 13 LLMs, the best model achieves 9.6% Macro Defect Recovery Rate on real-world instances but 80.6% Macro Clarification Action Success Rate when given the defect. In an oracle study, supplying the gold resolution raises the downstream codification-ready rate from 14% to 98%. Across all evaluated models, defect localization is the main bottleneck, with stronger clarification given the defect.