Multimodal model uses annotator views to detect online sexism
Through the Eyes of the Beholder: Biometric and Demographic Conditioning for Multimodal Sexism Detection
Computation and LanguageArtificial IntelligenceComputer Vision and Pattern RecognitionMachine Learning
Summary
Detecting sexism online is tricky because people can have different opinions on what counts as sexist. The authors made a computer system that looks at memes using images, text, and other signals, and also considers the backgrounds and opinions of people who labeled the data. Their approach tries to learn from the full range of opinions, not just a single label. This method helps make predictions that reflect human subjectivity better than standard models.
What this means in practice
- •For content moderation teams: Enhance online sexism detection systems by integrating annotator background information to better capture subjective interpretations of harmful content.
- •For social media platform developers: Build multimodal tools that combine image, text, and user annotation perspectives for more nuanced understanding of online hate and bias.
Authors
Ana-Maria Luisa Mocanu, Sebastian Mocanu, Ciprian-Octavian Truică, Elena-Simona Apostol
Abstract
Detecting sexism on the internet is a fundamentally subjective task; our team, VANGUARD, addresses this challenge in the EXIST 2026 Task 2 by proposing a human-centered multimodal framework that analyses and incorporates the psychological and demographic characteristics of human annotators into the detection pipeline. We fuse five input modalities through a cross-attention architecture with Feature-wise Linear Modulation conditioning. Meme text is extracted and visually described with Gemma 4, then augmented by automatic translation between English and Spanish with NLLB-200. Text and image representations are produced by LoRAadapted XLM-RoBERTa and CLIP encoders and fused with sensor features encoded by a pretrained autoencoder. To model annotator subjectivity, we frame Subtask 2.1 as a label distribution learning problem, optimizing a Kullback-Leibler divergence loss over the full annotator label distribution. At inference time, predictions are produced by soft-voting between the deep multimodal network and a complementary SVM trained on stylometric and physiological features. Our best submission ranks 29th out of 114 on Subtask 2.2 (source intention) under soft evaluation, and the normalized ICM scores remain above the baseline on Subtasks 2.1 and 2.2, indicating that annotator-centered conditioning contributes a usable signal. We release our full pipeline and analysis to support reproducible human-centered modeling.