Detecting implied arguments in political tweets faces disagreement challenges
Interpretive Variation and Discourse Structure in Detecting Enthymemes in Controversial Political Discourse
Computation and Language
Summary
The paper studies arguments called enthymemes, where some parts are left out, so readers have to guess them. This guessing can lead to disagreement on whether an argument is present. The authors trained computer models to detect these in tweets about vaccines and immigration, but the models often scored no better than 0.67 F1, showing difficulty in agreement. They suggest new ways to measure success by considering different readers’ opinions and show models rely on clear argumentative clues but miss hidden connections. This raises caution about using such methods for moderating online content.
What this means in practice
- •For content moderation teams: Improve evaluation of argument detection models by incorporating annotator disagreement for more nuanced moderation decisions.
- •For social media platform engineers: Design moderation tools that better handle hidden or implied arguments often missed due to implicit discourse links.
Tested on one dataset.
Authors
Martial Pastor, Nelleke Oostdijk
Abstract
Enthymemes are arguments that leave a premise or conclusion unstated, and they are a common persuasion device on social media. Because readers must fill in the missing part themselves, they often disagree on whether a text contains one. This paper asks how to detect and evaluate enthymemes when such disagreement is part of the task. On tweets about vaccination and immigration, each labeled by five annotators, F1 stalls around 0.67 for every fine-tuned model we train, regardless of data size, added features or label type. We therefore ask how these models should be evaluated instead, and propose metrics that compare predictions with the full distribution of annotator votes, together with a model that predicts this distribution directly and captures disagreement better at no cost in F1. We then ask what role discourse structure plays. Models learn to rely on explicit argumentative structure: they reliably recognize tweets without an enthymeme but rarely commit on tweets that contain one. Both the models and a discourse parser miss arguments whose links are left implicit and must be inferred by the reader. Finally, we discuss what these blind spots mean for content moderation systems trained on majority labels.