ReScraper improves web data cleaning for better large language models
ReScraper: Unified Scraping and Cleaning of Web Data for Effective LLM Pretraining
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
Training large language models requires lots of clean text from the internet, but current cleaning uses many hand-coded rules that can miss problems. The authors created ReScraper, a small AI model that learns to find the main content on web pages and fix or remove messy parts automatically. This unified approach improves the quality of the training data more effectively than previous multi-step methods. Their tests show smarter cleaning helps models learn better language skills without losing variety in the data.
What this means in practice
- •For machine learning engineers: Improve the quality of training data for language models by replacing heuristic filters with a learned model that extracts and cleans web pages in one step.
- •For data pipeline developers: Simplify data preparation pipelines by consolidating scraping and cleaning tasks into a single AI model that adapts operations based on page quality.
Authors
Zichun Yu, Jiarui Yan, Shlok Sanghvi, Nihar Atri, Chenyan Xiong
Abstract
LLM pretraining corpora are normally cleaned by a stack of hand-written heuristics. A heuristic scraper extracts the main content from HTML, and dozens of rule-based filters then clean it, so corpus quality is capped by the coarseness and accuracy of the rules. In this work, we propose ReScraper, a unified language model of only 0.6B parameters that replaces this entire stack. To train ReScraper, we carefully curate supervised data from the outputs of three teacher models, so it learns to first extract the main content from raw data and then choose among four operations: keeping the page as extracted, editing out noisy lines and spans, deleting it entirely, or rewriting it when it is poorly written but informative. Based on the same crawled data pool, pretraining 400M, 1.4B, and 2.8B models on our curated data improves the DCLM Core score by a relative 3.8--4.7% over the strongest baseline at each scale, including the costly multi-agent curation. Our analyses show that each operation plays a distinct and complementary role, and that extracting and cleaning in one model outperforms a cascade of separate models. ReScraper also concentrates its operations on the pages that need them, raising the quality of poor pages the most while keeping the corpus diverse. These results demonstrate the feasibility and effectiveness of AI4AI for pretraining data curation, where a small learned model takes over an entire stage of the pipeline from hand-written heuristics. We open-source our code at https://github.com/cxcscmu/ReScraper