URL extraction from scholarly papers varies by document format
URL Extraction from Scholarly Documents: A Cross-Format Comparative Analysis
Digital Libraries
Summary
Many research papers include URLs linking to useful materials like datasets and software. Extracting these links correctly is important for various tasks but previous work often looked at only one document type, usually plain text. These authors studied how well URLs can be extracted from six different document formats and combinations using a large set of annotated scientific papers. They found that some formats and format combinations work much better for finding URLs. They also tracked how URL usage and extraction changed over 33 years of research papers.
What this means in practice
- •For digital librarians: Improve harvesting of URLs from various document formats to create more accurate and comprehensive archives of research links.
- •For data engineers: Build better pipelines for extracting URLs from scholarly documents using recommended format combinations to support data integration and analysis.
Authors
Rochana R. Obadage, Lamia Salsabil, Sawood Alam, Bipasha Banarjee, William A. Ingram, Edward A. Fox, Jian Wu
Abstract
URLs in scholarly documents link to rich external resources such as datasets, software, publications, and websites. Extracting these URLs is crucial in the data preparation stage of many downstream tasks, such as link rot analysis, web crawling, and building knowledge graphs. However, existing studies often downplay this phase, simply extracting URLs from a single format, usually text directly converted from PDFs. We present a systematic study evaluating URL extraction across six input formats (text with annotation layer, LaTeX, HTML, XML, Markdown, and PNG converted from PDF). To support the evaluation, we compiled a benchmark dataset consisting of 2,338 manually annotated URLs from 200 arXiv papers spanning a wide range of domains over a 33-year period. In addition to evaluating individual file formats, we also compared 63 composite input-format combinations. Our extensive evaluations indicate that TEXTWAL achieves the best performance among single-format inputs, while TEXTWAL+LaTeX achieves the best overall URL extraction performance. The same trend is observed for URLs linking to open-access datasets and software. To further validate these findings, we apply our format-specific URL extraction pipelines to a longitudinal random sample of 364,744 arXiv papers spanning 33 years. We observe a sharp increase in URL density after 2015, along with remarkable differences in URL extraction across file formats over time. Overall, our study highlights the importance of selecting an appropriate format for URL extraction from scholarly documents. The dataset and code are publicly available at: https://github.com/lamps-lab/arxiv-url-bench .