Large language models detect hidden stories in social messages without training
Zero-shot narrative detection in social messaging
Computation and Language
Summary
Finding hidden stories or messages in social media posts is hard, especially without prior examples to learn from. The authors show that big language AI models can spot these hidden narratives by using human written descriptions, even without training on those examples. When the descriptions are generated automatically or few examples are used, performance drops. Combining several model outputs improves results, and bigger models work best and are less sensitive to instructions. This suggests a new way to identify complex message meanings in many areas without needing lots of training data.
What this means in practice
- •For social media monitoring teams: Identify hidden strategic narratives in public social messages without needing training data for each new domain.
- •For online content moderation teams: Detect covert or subtle narratives in harmful messages quickly using large models and human-written narrative guides.
- •For marketing analytics departments: Discover underlying customer stories in social conversations without retraining models for every campaign or topic.$Commercial implications: Enables development of scalable narrative detection tools for brand reputation management using zero-shot large language models.
Authors
Jesús M. Fraile-Hernández, Anselmo Peñas, Patrick Giedemann
Abstract
This study investigates the zero-shot ability of large language models (LLMs) to identify and classify hidden narratives in social messages. Our research hypothesis is that LLMs' extensive contextual knowledge allows them to interpret messages on a deeper, pragmatic level, going beyond basic sentiment or topic analysis. Experiments on the Dipromats and SemEval datasets show that providing models with human-written narrative descriptions significantly improves performance, without the need of training examples. In contrast, automatically generated descriptions or the use of few examples (few-shot) often degrade accuracy due to subtle shifts in framing. The study also finds that ensemble methods, particularly majority voting, enhance robustness and that larger models perform best while also being less sensitive to prompt variations. The findings validate that LLMs can effectively detect strategic narratives in a zero-shot setting, and when combined with simple ensembling and human-written descriptions, they can rival supervised systems, offering a scalable solution for narrative detection, specially when there is no training data for the vast majority of domains.