European Portuguese web corpus expands with refined filtering methods
Fine PT-PT Web: A High-Quality 41 Billion Tokens Data Collection of the European Portuguese Web
Computation and LanguageArtificial Intelligence
Summary
Collecting online text for specific language versions like European Portuguese is tough because it can get mixed up with Brazilian Portuguese and involves huge amounts of data. The authors created a smart process that cleans and sorts web data early to save more good content that usually gets thrown away. Their method also uses careful language checks, fuzzy duplicate removal, and AI-based quality checks to produce a clean and useful text collection. This collection can help train large language AI models specifically for European Portuguese.
What this means in practice
- •For language technology developers: Use the curated European Portuguese corpus to train language models that better understand regional language variation.
- •For natural language processing teams: Improve text processing tools for European Portuguese by incorporating the clean, large-scale web data collection.
Authors
Gonçalo Vinagre, Rui Pedro Guerra, Pedro Gomes, Miguel Moura Ramos, Duarte Miguel Alves, Afonso Simplício, Diogo Tavares, David Semedo, Daniel Gomes, João Magalhães
Abstract
Curating Web corpora for regional language variants like European Portuguese (PT-PT) is heavily bottlenecked by dialectal overlap (mainly with PT-BR) and data processing scale. This paper presents an efficient pipeline to curate a production-ready PT-PT corpus from the Portuguese Web, spanning 411 TB of raw data from Arquivo.pt. We introduce a novel post-scraping block that removes boilerplate and line duplicates prior to filtering. This early-stage intervention increases final document yield by 19.04% by rescuing valid text that standard heuristic filters prematurely discard. Integrated with rigorous language identification, weighted fuzzy deduplication, and neural quality classification, our pipeline offers a scalable framework and a clean, representative corpus optimized for LLM pre-training.