Counting Documents Is Not Counting Text: Unit Bias in Web-PDF Corpus Statistics
2026-08-17 • Computation and Language
Computation and LanguageArtificial Intelligence
AI summaryⓘ
The authors studied a large collection of web PDFs and found that just a small number of documents contain most of the words. Many long PDFs take up a lot of the total text, and those created with TeX are a notable subset. They discovered that a size limit causes many PDFs to be cut off, losing a large part of their text. Attempts to recover this lost text only succeed in a small fraction of cases. The authors suggest that statistics about such collections should report both the number of documents and the number of tokens to give a clearer picture.
tokensPDF corpusCommon Crawltext truncationOCRTeXdata recoveryGini coefficientdocument lengthcorpus statistics
Authors
Luca Foppiano
Abstract
PDF corpora advertise their size in tokens but compute every rate they publish (coverage, OCR routing, re-fetch recovery, language mix) per document, and none decomposes its token total. The two units diverge sharply. On CC-MAIN-2021-31-PDF-UNTRUNCATED (7.9M web PDFs, 32.6B tokens), 3.02% of text-bearing documents hold half the tokens (Gini 0.807); documents over 50 pages are 5.00% of the corpus but 53.53% of its text. The PDFs produced by a TeX{} toolchain are 1.66% of documents and 4.05% of the text. The clearest casualty is Common Crawl's truncation cap: it affected 23.06% of documents and 63.08% of the text. Reconstructing the truncated files and extracting both versions, two widely used libraries recover 11.4% and 1.4% of that text; between 72% and 97% of affected documents yield nothing; roughly 55--62% of the corpus's text is lost. Under the 5 MiB cap adopted in March 2025, 30.19% of tokens would still be truncated, and recovery on those documents rises only from 3.3% to 13.2%. We recommend that corpus statistics be reported in both units: documents and tokens.