Beyond the Literal: Decomposing Pragmatic Intent in Multimodal Meme Understanding
2026-06-02 • Computation and Language
Computation and Language
AI summaryⓘ
The authors found that big vision-language models often explain memes by just describing the picture, missing the real meaning or joke behind it. They created a method called Intent Projection that helps the model separate the literal content (what you see) from the intended message (what the author means). This method uses special techniques to remove obvious visual clues and focuses on the subtler, intended meaning. Their approach improved understanding on tests, especially when the meaning was very different from the literal image.
Large Vision Language Modelsmeme understandingliteral-pragmatic decompositionorthogonal projectionmultimodal benchmarkscontrastive learningsurface-real affect classifierstructured reasoninginstruction tuning
Authors
Zhengyi Zhao, Shubo Zhang, Zezhong Wang, Luyao Ye, Huimin Wang, Hanqi Yan, Binyang Li, Kam-Fai Wong, Yulan He
Abstract
When asked what a meme or sarcastic post means, Large Vision Language Models (LVLMs) tend to describe what the image shows rather than what the author is trying to communicate. Standard instruction tuning entangles a post's literal content with its pragmatic meaning, letting surface-level details contaminate the final response. We reframe meme understanding as a problem of literal-pragmatic decomposition and propose \textbf{Intent Projection}, a framework that separates the two signals at the representation, output, and objective levels within a single LVLM backbone. At the representation level, an orthogonal projection module removes dominant unimodal directions from the fused image-text representation, retaining only the pragmatic residual, while a surface-real affect classifier anchors the decoder with a discrete tag that names the polarity gap. At the output level, the model externalizes a structured reasoning chain, and at the objective level a contrastive reward explicitly penalizes answers that restate the literal description. Across six multimodal benchmarks, Intent Projection consistently outperforms open-source baselines and narrows the gap to proprietary models, with the largest gains on high-divergence posts where literal collapse is most damaging.