Community archive offers ethically sourced dataset of 5 million social posts
The Community Archive: An Ethically Sourced X (Twitter) Dataset
Social and Information Networks
Summary
Many social media datasets are collected without asking users for permission, which raises privacy and ethical concerns. The authors created The Community Archive, a large dataset of over 5 million posts from X (formerly Twitter) where each user gave consent before their content was included. This allows researchers to access both user posts and related data ethically and at a scale that was not possible before. They showed how this dataset can be used to study how similar people connect in social networks.
What this means in practice
- •For social media analysts: Analyze user connections and content patterns using a consent-based dataset of 5 million X posts.
- •For natural language engineers: Train language models on ethically sourced, large-scale social media text data with metadata included.
Authors
Brandon Duderstadt, Omar Shehata, Ivan Vendrov, Francisco Carvalho
Abstract
We present The Community Archive, a novel dataset of over 5 million posts from the social media site X (previously known as Twitter). In contrast with other large scale social media datasets, which are generally assembled without user consent via web scrapers, The Community Archive uses an opt-in model to obtain author consent for every post in the dataset. This enables us to release both the metadata and the content of users' posts at a previously unprecedented scale. Our dataset supports work in a wide variety of domains, including social network analysis, narrative tracking, and diachronic language modeling. We demonstrate the utility of our dataset by conducting a sample analysis of content-based homophily in our network.