AI summaryⓘ
The authors studied how large language models (LLMs) detect antisemitism by using different types of background information about the concept. They found that detailed category-based information helps the models find more antisemitic cases but also makes them less precise. Surprisingly, giving the models much bigger sets of information didn’t improve their performance. They also discovered that models struggle the most with detecting antisemitism that appeared after the Holocaust and tend to rely too much on specific words or explanations that can be overconfident or miss subtle forms. Overall, the authors show that while concept-based grounding helps, LLMs still face important challenges in understanding antisemitism fully.
Large Language ModelsAntisemitism DetectionConceptual GroundingRecall and PrecisionTaxonomic RepresentationPost-Holocaust AntisemitismInference TimeExpert-Annotated DatasetsLexical CuesModel Explainability
Authors
Katharina Soemer, Helena Mihaljević
Abstract
LLMs enable the integration of external conceptual resources at inference time, creating new opportunities for detecting ideologically and historically complex phenomena such as antisemitism. We investigate how different forms of conceptual grounding affect antisemitism detection and explanation behavior across four state-of-the-art LLMs. Using two expert-annotated datasets, we compare definitional, fine-grained taxonomic, example-augmented, and large-context representations of antisemitism. We find that fine-grained taxonomic representations substantially improve recall, while simultaneously reducing precision. Surprisingly, supplying substantially larger conceptual resources yields no additional quantitative benefit. Post-Holocaust antisemitism poses the most persistent challenge across models and configurations. Analysis of explanations further reveals systematic limitations including overproduction of conceptual references, reliance on lexical cues, overconfidence, and difficulties with subtle or justificatory forms of antisemitism. Our findings highlight both the potential and the remaining limitations of conceptually grounded LLMs for antisemitism detection and reasoning.