Culture language and region annotations enhance web data benchmarking
FineWeb-CLaR: Culture, Language, and Region Annotations for Benchmark-Aligned Corpus Auditing
Computation and Language
Summary
It’s hard to tell if language AI models understand different cultures because the training data and tests are labeled differently. The authors created FineWeb-CLaR, a big dataset that tags web documents with culture, language, and region information, making it easier to compare what’s in the training data with what tests measure. They also labeled many benchmarks to match this system, so it’s clearer if models truly cover cultural topics. This helps check if AI learns from or is tested on diverse cultural content.
What this means in practice
- •For nlp engineers: Identify gaps in cultural and regional content coverage in AI training data to improve model fairness and relevance.
- •For benchmark developers: Design better cultural benchmarks for language models by aligning test data with pretraining corpus metadata on culture and region.
Authors
Yusser Al Ghussin, Eva Gavaller, Cristina España-Bonet, Josef van Genabith, Simon Ostermann
Abstract
Cultural evaluation coverage and robustness in language models are difficult to diagnose because pretraining corpora and cultural benchmarks are rarely indexed with comparable metadata. Benchmarks increasingly target culturally situated phenomena at the level of languages, regions, and locale-specific practices, while web-scale corpora are usually organized only by language. A shared culture-language-region layer makes these resources comparable, enabling audits of whether a target cultural phenomenon is represented in pretraining data, evaluated by benchmarks or both. To this end, we introduce FineWeb-CLaR, a large-scale annotated dataset derived from FineWeb and FineWeb-2 that places web documents on a shared culture-language-region axis for corpus auditing and benchmark alignment. FineWeb-CLaR annotates the full 30.9B-document collection from FineWeb and FineWeb-2 with URL-derived region labels and cultural-topic provenance. Our region resolver assigns a non-empty region to 25.61% of documents (7.92B). For cultural-topic analysis, we induce locale-specific topics and project them onto the 14 leaves of the Cultural Taxonomy of Liu et al. (2025), producing Locale Topic Distributions (LTDs) for corpus-side comparison. We also annotate 277 cultural NLP benchmarks with the same taxonomy, language coverage, and region coverage. Together, these resources enable direct comparison between corpus-side pretraining evidence and benchmark-side evaluation coverage.