Unveiling the Limits of Large Language Models in Inferring Pragmatic Meaning from Non-Verbal Responses
2026-06-01 • Computation and Language
Computation and LanguageArtificial Intelligence
AI summaryⓘ
The authors studied how well large language models (LLMs) understand meaning when people communicate using only non-verbal actions, like gestures, instead of words. They found that LLMs have a hard time figuring out indirect messages from non-verbal responses, doing much worse than when interpreting spoken language. The researchers also looked into why these mistakes happen and showed that teaching LLMs with examples in context can help them understand non-verbal intent better.
large language modelspragmatic language understandingnon-verbal communicationindirect meaningdialoguein-context learningpragmatic inferenceverbal behaviormachine learningnatural language processing
Authors
Sugyeong Eo, Heuiseok Lim
Abstract
Although large language models (LLMs) have shown considerable progress in pragmatic language understanding, prior research has focused mainly on their comprehension of verbal behavior. Nonetheless, non-verbal behavior remains a fundamental component of human communication, especially when deliberately utilized in isolation to convey indirect meanings. In this work, we present the first systematic evaluation of LLMs' ability to infer pragmatic meaning in dialogue consisting solely of non-verbal responses. We explore three research questions: (1) Can LLMs recognize indirect intent conveyed through non-verbal responses? (2) When and how do LLMs fail to capture non-verbal intent? (3) How can we improve LLMs' ability to interpret non-verbal intent?. Through the evaluation, we observe that LLMs struggle to infer underlying meaning from non-verbal responses, with accuracy dropping by up to 60% points compared to verbal ones. Further extensive analysis reveals a behavioral pattern in LLMs' interpretations of non-verbal behavior and demonstrates that in-context learning facilitates pragmatic inference.