Llms fail to handle character attacks in political debates effectively
Benchmarking Argumentative Behaviour of LLMs: A Study of Defences Against Character Attacks
Computation and Language
Summary
This paper shows that large language models (LLMs) are not as good as humans at defending themselves when people attack their character during political arguments. While humans often respond to these personal attacks using strategic moves that involve character and reputation, LLMs mostly stick to logical defenses and ignore these kinds of responses. The authors analyzed real political debates and found that safety rules might limit LLMs from fully engaging in these kinds of complex, realistic conversations.
What this means in practice
- •For dialogue system developers: Improve argumentation models by incorporating defense strategies against character attacks identified in political dialogues.
- •For political campaign teams: Design more human-like AI agents that engage in political debate simulations reflecting realistic personal attack defenses.
Authors
Ewelina Gajewska, Katarzyna Budzynska, Jaroslaw Chudziak
Abstract
Large Language Models (LLMs) are increasingly deployed as argumentative agents in persuasive dialogues, necessitating rigorous evaluation of their debating competence relative to human interlocutors. In this study, we focus on character attacks (ad hominem arguments), traditionally dismissed as fallacies, which play a pivotal role in political persuasive dialogues where ethos often rivals propositional content. Specifically, we investigate whether modern LLMs can replicate human competence to strategically use and respond to such attacks. We analyse a corpus of natural language political dialogues to identify defensive strategies human interlocutors naturally employ in ethos-centred debates and structure them into a dialogue game. Empirically, we benchmark LLM-generated dialogues against the ElecDeb60to16-fallacy corpus of U.S. presidential debates, contrasting human debaters' repertoire of defensive strategies with those of artificial agents. Results reveal a substantial difference: most LLMs rigidly prioritise logical defences, failing to exploit ethotic counterattacks as valid moves in political discourse. We argue that current safety fine-tuning constraints the strategic action space of these LLMs, making them unable to fully engage in naturalistic interactions within domains where character contestation is a normative expectation rather than a mere fallacy.