Summary
Identifying harmful memes can be tricky for AI models if they either miss clues inside the images or fail to use these clues correctly when deciding if the meme is harmful. The authors studied two big AI models and found that the problem often lies in how the models direct or 'route' the important information to their decision parts, not in missing the information itself. They showed that by changing how the models pick up and use this information, the models do much better at spotting harmful memes across different tasks and languages. This means that improving the way AI channels evidence could make harmful meme detection more reliable.
What this means in practice
- •For social media platform engineers: Improve harmful meme detectors by modifying model routing to better utilize visual and textual clues, enhancing detection accuracy across languages.
- •For multilingual content moderation teams: Use enhanced AI models that reliably detect harmful memes in English, Spanish, and code-mixed Hindi-English content by fixing feature routing bottlenecks.
Abstract
When a large vision-language model misclassifies a harmful meme, the failure may reflect missing internal evidence or an inability to route represented evidence to its output. We distinguish these cases in Gemma-3 and Qwen3.5 using sparse autoencoders, role-conditioned probes, causal interventions, and recovery experiments across six harmful content benchmarks, with additional Spanish and Hindi-English code-mixed evaluations. Sparse readouts outperform native prediction on all six primary binary tasks: Qwen averages $0.740$ versus $0.432$ native macro-F1, while residual reconstruction reaches $0.486$, whereas Gemma improves from $0.532$ to $0.714$. These differences reflect supervised accessibility rather than a pre-existing, native decision rule, and the most influential token role depends on the task. Under the evaluated score scales, Qwen silent-feature ablation is $24-63$ times more probe-sensitive, whereas routed-feature patching on literal yes/no tasks is $16-140$ times more output-sensitive. Calibration-only routing recovers $93.3$% of the mean gap, and probe-distilled LoRA improves native predictions, although shared multi-task adaptation causes negative transfer. A case study of Gemma-3-12B on Facebook Hateful Memes finds a distributed rank-32 image-prompt interaction, reaching $0.756$ versus $0.685$ native macro-F1. Robustness controls show that the signal extends beyond English, is not explained solely by accompanying OCR, and depends on paired visual evidence. Thus, routing, rather than representation alone, is a recurring bottleneck in harmful meme classification.