The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers
2026-06-15 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors studied how AI research papers have improved their documentation over the last decade to help make their work easier to check and reuse. By analyzing nearly 57,000 papers from top AI conferences, they found that the sharing of code and data grew from 11% in 2014 to 64% in 2024. This suggests that reproducibility—the ability to get the same results from the same data and code—also improved, nearly doubling in that time. They noticed these positive changes began even before official checklists were introduced, indicating a wider shift toward openness in AI research.
reproducibilitydocumentation practicesAI conferencescode sharingdata sharingopen sciencereproducibility checklistsmethodological issues
Authors
Kevin L Coakley, Thijs Snelleman, Holger Hoos, Odd Erik Gundersen
Abstract
The reproducibility crisis has directed the AI research community toward improving documentation practices. Several studies have identified methodological issues, and in response, the most impactful venues in the field have introduced reproducibility checklists. We seek to understand whether documentation practices have changed over time by assessing all published papers at five leading AI conferences over the past decade. Seven reproducibility variables were identified, quality-assured and used to analyse 56 800 publications. Our analysis reveals that in the period 2014 to 2024, documentation practices have improved; papers sharing both code and data increased nearly sixfold, from 11% to 64% Building on empirical reproducibility rates from a prior study, we estimate - inferred from documentation practices, not direct testing - that reproducibility increased from 28% in 2014 to 64% in 2024. Improvements in documentation practices predate the introduction of reproducibility checklists, suggesting these changes reflect a broader movement toward open science rather than a direct response to formal requirements.