Unsupervised methods reveal hidden disinformation narratives in text datasets
From Repetition to Recognition: Inductive Discovery of Disinformation Narratives
Computation and Language
Summary
Disinformation often spreads through repeated stories or themes called narratives. The authors studied how to automatically find these narratives without relying on existing lists or labels. They tested methods that group texts by similarity or connections between ideas on multiple datasets. Their findings show that some methods better find balanced sets of narratives, including rare stories that others miss. This can help catch misleading information patterns people might not know yet.
What this means in practice
- •For content moderation teams: Identify emerging disinformation themes without needing predefined labels to improve monitoring of harmful narratives.
- •For social media analysts: Use graph-based and clustering methods to detect lesser-known or rare disinformation stories within large text collections.
Authors
Max Upravitelev, Veronika Solopova, Jing Yang, Charlott Jakob, Alexandra Tsiakalou, Neda Foroutan, Vera Schmitt
Abstract
In disinformation datasets, narratives are often understood as recurring interpretive patterns that group texts under narrative labels. Recent work formalized narrative mining as inductively inferring narrative labels from corpora, but its evaluation stays tied to predefined taxonomies, a closed-world setting that cannot capture narratives absent from the reference labels. We introduce a three-tier evaluation framework for unsupervised narrative label generation: recovery (against a corpus's own taxonomy), mining (against external label sets), and discovery (without predefined labels). Applying it, we compare clustering-based and graph-community-based pipelines across seven disinformation datasets, with human validation of discovery on two. The two families are complementary under automated metrics, but in a corpus with two prominent topics, clustering can reduce one topic to 2% of generated labels while graph-based pipelines stay balanced. Discovery validation also reveals many singletons (narrative labels derived from single claims, 30-62% of graph outputs), which clustering cannot produce. Annotators confirm many as recognizable disinformation narratives, suggesting that in open-world discovery the repetition assumed by narrative mining may be recognized outside the corpus, not within it. We release human-validated narrative candidate labels for the Climate Obstruction and PolyNarrative datasets to support taxonomy development and dataset extension.