PragMatch: Separating Pragmatic Incongruity from Cross-Modal Mismatch in Large Vision-Language Models
2026-08-10 • Computation and Language
Computation and Language
AI summaryⓘ
The authors studied whether large vision-language AI models truly understand the relationship between pictures and text, especially in tricky cases like sarcasm. They created a special test set called PragMatch with examples where sarcasm depends on subtle clues, not just obvious mismatches. Their experiments showed that these models often rely on simple surface cues like word style or visible text rather than deeper understanding. This suggests the models have limits in recognizing sarcasm that needs reasoning beyond basic matching.
Large Vision-Language ModelsMultimodal Sarcasm DetectionPragmatic IncongruityShortcut LearningOCRLexical CuesSurface-Level AlignmentMMSD2.0BenchmarkMultimodal Reasoning
Authors
Zhanna Mukhametsharip, Vera Demberg, Varsha Suresh
Abstract
Large Vision-Language Models (LVLMs) have demonstrated strong performance on multimodal benchmarks, yet it remains unclear whether they genuinely reason about relationships between images and text or rely on superficial correlations, known as shortcut learning. This question is particularly important for multimodal sarcasm detection, where successful prediction depends on recognizing pragmatic incongruity rather than treating sarcasm as simple image-text mismatch. We introduce PragMatch, a controlled benchmark of 3,000 image-text pairs derived from MMSD2.0, including original sarcastic examples and constructed literal and hard-negative pairs. We identify influential shortcut cues through systematic masking and evaluate their impact through targeted injection experiments. Our results show that LVLM predictions are sensitive to lexical, OCR-derived and stylistic cues, with injected surface signals causing substantial changes in model predictions despite unchanged underlying image-text relationships. Our findings reveal limitations in current LVLMs while PragMatch provides a systematic testbed for evaluating multimodal pragmatic reasoning beyond surface-level image-text alignment.