When Models Hear What They Expect: Diagnosing Prosodic Heuristics in Multimodal Sarcasm Detection
2026-08-31 • Computation and Language
Computation and Language
AI summaryⓘ
The authors studied how multimodal large language models (MLLMs), which process both speech and text, handle sarcasm detection. They tested models on Mandarin and English using different inputs to separate the effects of words, meaning, and speech patterns. They found that adding audio caused the models to make more wrong guesses by relying on stereotyped speech cues like high pitch and irregular pauses, which don't actually signal sarcasm in these languages. This issue appeared in multiple models, suggesting it is a common problem rather than one specific to a single model type.
multimodal large language modelssarcasm detectionprosodyacoustic featuresfalse positivespitchpausingspoken language processingpragmatic inference
Authors
Yongjian Chen, Pengfei Wei, Yiqun Sun, Zhu Li, Lawrence B. Hsieh
Abstract
Multimodal Large Language Models (MLLMs) process speech and text jointly, yet whether they exploit prosodic cues for pragmatic inference or rely on surface acoustic patterns has received little systematic investigation. We address this through sarcasm detection, evaluating Qwen2.5-Omni and Qwen3-Omni on Mandarin Chinese and English under five modality conditions that decompose the contributions of lexical content, vocal semantics, and prosodic structure. Adding audio systematically inflates false positives without improving true positive detection. Acoustic error diagnosis reveals that model errors cluster on a shared stereotype of expressive prosody, namely elevated pitch and irregular pausing, that diverges from the actual cues marking sarcasm in both languages. Targeted manipulation of only these two dimensions causally confirms the heuristic, inducing false positive rates of up to 60%. Applying the same manipulation template to Gemini~3 Flash Preview without modification replicates the effect, suggesting that the stereotype extends beyond the Qwen Omni family rather than arising from a single model architecture.