Github code linked in linguistics papers often disappears over time
Papers Without Code: Availability of GitHub Repositories Linked in *CL Publications
Computation and Language
Summary
Many papers in computational linguistics share their code on GitHub, which helps others use and check their work. However, the authors found that links to GitHub code in recent papers disappear just as often as in older papers. This is partly because some repositories are empty or just placeholders. Other platforms don’t show this problem as much.
What this means in practice
- •For open source project managers: Monitor and maintain linked GitHub repositories to ensure continued access to code referenced in research outputs.
- •For digital archiving teams: Develop strategies to archive research code repositories to prevent data loss from empty or removed GitHub links.
Authors
Selina Meyer, Michael Roth
Abstract
Source code and data published at computational linguistics (*CL) venues are increasingly being shared via GitHub. While this generally is a favourable development for the accessibility and potential reusability of research artifacts in natural language processing (NLP), the long-term availability of such repositories has not been evaluated. In this squib, we discuss the availability of repositories linked in papers published in the Computational Linguistics (CL) journal as well as at ACL and its co-located events over the past ten years. Contrary to our expectations, we find that GitHub repositories linked in more recent ACL publications are unavailable at similar rates as in older publications, in parts due to an increase in empty and placeholder repositories. Similar trends hold for other *CL venues, but not for platforms other than GitHub.