Definitional Sensitivity in Media Bias Detection: A Multi-Definition Dataset and Benchmark
2026-08-24 • Computation and Language
Computation and Language
AI summaryⓘ
The authors studied how different ways of defining media bias affect how people and language models identify bias in news articles. They found that when definitions change in their main focus, it causes large differences in how bias is labeled, especially for AI models. However, simply adding extra details to the same main definition did not cause big differences. The authors highlight that it is important to be clear and consistent about what bias means when creating datasets or AI tools. They also created and shared a new dataset called MUDD for studying bias detection with multiple definitions.
media biasbias detectionannotationdefinition framinglarge language models (LLMs)media bias datasetsconstruct specificationprompt-based measurementhuman annotationMUDD dataset
Authors
Martin Wessel, Timo Spinde, Jürgen Pfeffer, Gianluca Demartini
Abstract
Media bias detection relies on definitions and examples that specify what counts as bias, yet these specifications often vary across datasets or remain implicit, even when given the same name. Such variation makes it unclear whether models trained for the same bias category learn the same construct or different phenomena, a problem largely overlooked in prior work. We examine how definition choice affects bias annotation in a between-subjects experiment with 354 participants and a parallel evaluation with four LLMs. Participants and models rate six news articles across four bias categories using definitions that vary in conceptual framing and elaboration. Across 8,496 human and 28,800 LLM ratings, we find that the conceptual target of a definition drives annotation divergence, while construct-preserving elaboration does not: conceptual framing significantly shifts annotations for humans and does so even more strongly for LLMs. We discuss implications for construct specification in annotation protocols and prompt-based measurement, and consider how definitional sensitivity may propagate to downstream classification beyond media bias. We also release MUDD, the Multi-Definition Bias Detection Dataset.