Evaluating Large Language Models for Antisemitic Incident Classification
2026-07-06 • Computation and Language
Computation and Language
AI summaryⓘ
The authors explore how AI, especially large language models like GPT-4o and Llama-3.2, can detect hateful events related to antisemitism from public reports. They found that these AI models show some ability to identify different types of hate incidents but still need significant improvements. Using clear definitions and example cases in the AI prompts helps the models perform better depending on the event type. They also tested the AI on college newspaper reports, showing potential for helping monitor hate early. The authors emphasize that cooperation between AI creators, lawmakers, and community groups is needed to improve these tools and fight hate effectively.
hateful event detectionlarge language modelsantisemitismGPT-4oLlama-3.2prompt engineeringfine-grained classificationhate monitoringin-context learningAI ethics
Authors
Karina Halevy, Julia Mendelsohn, Chan Young Park, Yulia Tsvetkov, Maarten Sap
Abstract
Addressing hate and violence in society requires timely detection of hateful events from public reporting, but automated identification of hateful events remains underexplored. We introduce the task of hateful event detection and investigate the ability of AI systems, specifically large language models (LLMs), to discover and classify reports of antisemitic events with fine-grained labels. We evaluate OpenAI's GPT-4o and Meta's Llama-3.2-3B-Instruct on multiple expert-annotated datasets containing antisemitic event descriptions from news articles, civil society reports, and official records. We show that LLMs, particularly GPT-4o, have potential for this task, but substantial improvement is needed. Providing clear term definitions and in-context examples in prompts can improve performance: definitions are most helpful for rhetoric-oriented events (e.g. classical antisemitic tropes), while examples help label action-oriented events (e.g. physical assault). A case study of college newspapers demonstrates that LLMs can help surface relevant real-world events, supporting early monitoring and intervention. Overall, our findings highlight both opportunities and critical gaps in AI's ability to recognize complex harms and underscore the need for collaborative efforts among AI developers, policymakers, and civil society to design models, implement robust evaluation, and develop policy frameworks for defining and combating hate efficiently and effectively.