European Portuguese web text harvested with new cleaning method
Fine PT-PT Web: A High-Quality 41 Billion Tokens Data Collection of the European Portuguese Web
Computation and LanguageArtificial Intelligence
Summary
Gathering internet text written in European Portuguese is hard because it often mixes with Brazilian Portuguese and includes repeated or irrelevant content. The authors developed a new way to clean and organize a huge amount of web data, carefully removing repeated lines and unnecessary parts earlier in the process. Their method saved a significant amount of useful text that other methods would have thrown away. This resulted in a large and high-quality collection of European Portuguese text that can help train language models better.
European PortuguesePortuguese BrazilianWeb corpusData cleaningDeduplicationLanguage identificationBoilerplate removalNeural quality classificationLarge language modelsData preprocessing
Authors
Gonçalo Vinagre, Rui Pedro Guerra, Pedro Gomes, Miguel Moura Ramos, Duarte Miguel Alves, Afonso Simplício, Diogo Tavares, David Semedo, Daniel Gomes, João Magalhães
Abstract
Curating Web corpora for regional language variants like European Portuguese (PT-PT) is heavily bottlenecked by dialectal overlap (mainly with PT-BR) and data processing scale. This paper presents an efficient pipeline to curate a production-ready PT-PT corpus from the Portuguese Web, spanning 411 TB of raw data from Arquivo.pt. We introduce a novel post-scraping block that removes boilerplate and line duplicates prior to filtering. This early-stage intervention increases final document yield by 19.04% by rescuing valid text that standard heuristic filters prematurely discard. Integrated with rigorous language identification, weighted fuzzy deduplication, and neural quality classification, our pipeline offers a scalable framework and a clean, representative corpus optimized for LLM pre-training.