Summary
Persuasive news writing often uses tricks that tap into mental shortcuts called cognitive biases to influence readers without them noticing. The authors created a dataset called BAIT containing news articles labeled with nine specific types of these bias triggers so models can learn to find them. They tested six advanced language models, which could mostly tell if an article had some bias triggers but often missed many and struggled to pinpoint the exact biased parts. The models also tended to flag common surface words rather than truly understanding the manipulative intent behind the text. This suggests these models aren't yet good at recognizing the real causes of bias in writing.
What this means in practice
- •For news platform developers: Build tools to flag likely biased content in news articles to improve reader awareness using the annotated triggers in BAIT.
- •For fact-checking teams: Create models that highlight specific biased passages to help human fact-checkers better target their reviews.
Abstract
Persuasive writing exploits cognitive biases, recruiting a reader's mental shortcuts to shape belief, often without the reader noticing. We call the linguistic features exploiting such mental shortcuts cognitive bias triggers. As large language models (LLMs) increasingly mediate how people encounter news, summarizing it, ranking it, and answering questions over it, a natural question is whether they can recognize the features of a text that bias a human reader. We introduce BAIT (Bias Annotation In Text), a resource for detecting cognitive bias triggers in news. BAIT pairs a schema of nine triggers, each specified in an annotator codebook, with 50 English news articles annotated by trained annotators at the article and span levels and adjudicated into a gold standard. On this data we define two tasks, detecting which triggers a text contains and locating the spans that carry them. We then benchmark six frontier LLMs zero-shot and with three or ten in-context examples. The models approach human agreement on which triggers an article contains, but rarely recover its full set of triggers, and they reach less than half of human agreement in locating the passages that carry them. Moreover, they over-predict triggers that come with easy surface cues, and most of the passages they mark for these triggers are not triggers, yet carry the trigger's surface cue far more often than random text. These findings suggest that the evaluated LLMs respond to surface cues rather than to the mechanisms that make a passage manipulative. We release the schema, the codebook, standoff annotations, and the evaluation pipeline.