Comparison shows key differences in global news data sources
Comparison of Common Crawl News & GDELT
Information Retrieval
Summary
Having a big collection of news stories from around the world helps computers understand what's happening everywhere. This paper looks at two popular collections of news called GDELT and Common Crawl News. The authors found that these collections get their news from different kinds of places—GDELT collects from broadcasts, printed media, and web news, while Common Crawl gathers news mainly by scanning many websites. This difference means they cover the news world in different ways.
What this means in practice
- •For data engineering teams: Choose the best news dataset for building news-driven AI pipelines by understanding their different source coverage.
- •For media monitoring organizations: Select appropriate news datasets to ensure more comprehensive or targeted event and trend tracking based on source types.
Authors
Ameir El Ouadi, David Beskow
Abstract
The corpus of worldwide news is important for natural language processing, knowledge graphs, large language models, and other technical efforts. Additionally, this corpus is important for understanding the people, places, organizations, and events that interact in real-time every day. This paper compares two news datasets used for these tasks today, namely the Global Database of Events, Language, and Tone (GDELT) and Common Crawl News. Our research highlights the strengths and limitations of each dataset, analyzing their content and coverage. Notably, while GDELT relies on broadcasts, prints, and web news from across the globe, Common Crawl focuses on news sites from around the world gathered through web crawling. Our analysis revealed considerable differences in where the two datasets gather their news sources.