Learning to Detect Cross-Modal Negation: An Analysis of Latent Representations and an Attention-Based Solution
2026-07-20 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors study how current AI models struggle to understand negation (like "not") when combining images and text. They find that standard vision-language models don’t clearly represent negation in their internal features. To fix this, the authors create a new model that better connects visual and language information, improving detection of negation by over 7% in tests. They also discover that negation in visuals often depends on the surrounding text, which is different from how negation usually works in language alone. Their approach helps computer systems better learn how negation works across different types of data over time.
NegationMultimodal learningVision-language modelsCross-modal attentionRepresentation learningSelf-supervised learningTemporal modelingJEPA2EmbeddingsF1 score
Authors
Ali AbuSaleh, Leon Hammerla, Alexander Mehler
Abstract
Detecting high-level semantic concepts like negation across modalities remains a challenge for current multimodal systems. We analyze this as a fundamental representation learning problem, providing the first evidence that negation does not form a linearly or non-linearly separable class in the latent spaces of standard vision-language models (VLMs). We demonstrate that pretrained embeddings primarily encode modality-specific features, lacking a generalizable negation signal. To overcome this, we propose a novel cross-modal attention architecture that explicitly models inter-modal dependencies, achieving performance gains of up to +7.03% F1 over unimodal baselines. Our analysis reveals a key asymmetry: while textual negation often appears independently, visual negation is semantically dependent on linguistic context, a finding validated through our statistical analysis of 3,222 political video-text pairs automatically annotated via \textsc{Qwen2.5-VL}. By combining this analysis with self-supervised video representations (JEPA2), we advance the modeling of temporal negation. This work provides new methods and insights for learning robust, semantically-aligned representations in multimodal systems.