RefinedGAW dataset boosts coverage of German academic web texts
RefinedGAW: A Refined German Academic Web Dataset
Computation and LanguageInformation Retrieval
Summary
Large collections of German text from the internet often miss specific academic and educational content. The authors created RefinedGAW, a huge dataset focused mainly on German academic texts gathered from many web crawls over time. This dataset has more academic content than similar collections and includes a detailed history and metadata for fine searching. It also has very few repeated texts compared to other datasets, offering fresh and diverse materials. The authors provide tools to access and use the documents effectively.
What this means in practice
- •For natural language processing teams: Use RefinedGAW to train language models focused on German academic and educational content with minimal commercial bias.
- •For digital library developers: Build search and retrieval tools leveraging the detailed metadata and queryable indexes provided to enhance academic content discovery.
Authors
Maximilian Idahl, Oleh Astappiev, Benedikt Droste, Robert Scholz, Björn Plüster, Jan Philipp Harries, Robert Jäschke, Wolfgang Nejdl
Abstract
The largest open German text datasets are built from broad web crawls, which provide scale but limited coverage of specific domains. We introduce RefinedGAW, a processed and annotated resource derived from 26 semi-annual crawls of the German Academic Web, totaling 122 million documents and 314 billion tokens, with full crawl and capture history of every text. Documents labeled as academic research account for 84% of German RefinedGAW tokens across HTML and PDF. At 87.6 billion tokens, this exceeds the 64.9 billion tokens in the German subsets of FineWeb2 and FinePDFs, despite their almost five-times-larger combined size. RefinedGAW also contains substantial amounts of educational, math-heavy, and expert-oriented content with minimal commercial bias. Across HTML and PDF, 88% of German RefinedGAW documents have no near-duplicate candidate match in the three evaluated reference datasets, supporting complementary coverage. We release a queryable metadata index for fine-grained data selection and a rehydration tool for retrieving and processing source documents.