Platform differences can skew machine-coded online policy sentiment
Is the Platform Part of the Measurement? A Protocol and Simulation Study for Cross-Platform Equivalence of Machine-Labelled Policy Sentiment
Machine LearningComputation and LanguageComputers and SocietySocial and Information Networks
Summary
Measuring people's opinions about policies using computers often involves looking at social media posts from platforms like Twitter or Reddit. This study shows that assuming the measurement works the same way on all platforms can cause mistakes, like getting the wrong overall mood. The authors created a step-by-step method to check if the sentiment scores from different platforms truly match or if they need adjusting. Their tests using computer simulations and real social media comments show that adjusting for platform differences leads to more accurate results.
What this means in practice
- •For social media analysts: Adjust sentiment scores by platform to produce more reliable cross-platform public opinion indices from social media data.
- •For market research teams: Use the protocol to verify if sentiment measures from multiple online sources can be combined without bias for product or policy feedback analysis.
Tested on simulated data.
Authors
Asad Ikram, Ella Haig
Abstract
Comparing machine-labelled sentiment across social media platforms is now routine: differences in classifier scores between platforms are read as differences in public opinion. These comparisons rest on an assumption that the classifier measures the same construct, on the same scale, on every platform. Some studies check a classifier's validity platform by platform, but to our knowledge none has modelled the platform as a facet of the classifier's measurement. We adapt the psychometric equivalence toolkit (generalizability theory, measurement invariance and differential item functioning, and score linking) to machine-labelled policy sentiment, treating the platform as a survey mode, and specify a six-step protocol with anchor events fixed in advance, an anchor-density audit, and a two-classifier convergence control. A simulation with known ground truth illustrates what the assumption can cost in one generating configuration fixed in advance: across 500 replications, because the platforms measure through different functions, a pooled index that takes platform scores at face value reports the wrong sign of aggregate sentiment on nearly a quarter of days, whereas an index adjusted with each platform's known measurement function does so on under one day in sixty on average. A case study of the instrument from the first author's MSc dissertation, a fine-tuned BERT classifier over 255,446 UK economic-policy comments from Twitter/X, Reddit, YouTube, and Quora, motivates the design and illustrates the instrument-dependence the protocol addresses. The full four-platform application, under fresh collection with the pre-specified design, is the designated follow-up study. The protocol lets a researcher test whether pooling platforms is defensible in a given corpus, or whether platform scores must be linked before any average is taken.