MetaphorVU: Towards Metaphorical Video Understanding
2026-05-25 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors created MetaphorVU-Bench, a first-ever test to see how well AI models understand metaphorical videos, which show ideas using comparisons. They found that current AI models have a hard time understanding these videos because they struggle to connect different ideas correctly. To help, the authors built a special knowledge graph and a new method called MetaphorBoost that improves the AI’s understanding during use. Their work offers a new way to study and improve AI’s ability to understand complex video content.
metaphorical videosmultimodal large language models (MLLMs)benchmarkcross-domain mappingknowledge graphinference-time enhancementMetaphorBoostvideo understandinghigh-order cognitive capabilities
Authors
Zhuoqun Li, Boxi Cao, Guiping Jiang, Fangrui Lv, Ruotong Pan, Jianan Wang, Xiangyu Wu, Hongyu Lin, Yaojie Lu, Yong Du, Ruyin Jia, Liyan, Tingting Gao, Han Li, Xianpei Han, Le Sun
Abstract
Metaphorical videos are prevalent across various real-world scenarios to convey complex ideas, and understanding them typically requires high-order cognitive capabilities. The lack of systematic studies on metaphorical video understanding not only constrains the real-world applicability of MLLMs but also impedes the thorough assessment of their high-order cognitive capabilities. To bridge this gap, we propose MetaphorVU-Bench, the first systematic and comprehensive benchmark dedicated to metaphorical video understanding. Through experiments, we find current MLLMs struggle with accurate metaphorical video understanding, lagging far behind human level, primarily due to defective cross-domain mapping. Motivated by this finding, we construct a metaphor knowledge graph as mapping augmentation and propose MetaphorBoost, an inference-time enhancement framework achieving consistent performance improvement. Our benchmark, analysis, and method provide useful insights and a foundation for future research on advancing MLLMs.