Vision language models struggle to judge duration of visits accurately
Leaving, Staying, or Living There? Duration Commonsense Reasoning with Human-LLM Co-annotations
Artificial IntelligenceInformation Retrieval
Summary
Figuring out how long someone stays somewhere from social media posts is hard because people rarely say the exact time. The authors created a new collection of tweets labeled by both humans and AI to test if current vision-language models can guess visit durations. They found the models often make mistakes, missing important details or focusing too much on obvious images instead of subtle words or hashtags. This shows current AI still has trouble understanding context well when estimating how long people stay in places.
What this means in practice
- •For urban planners: Improve models used to estimate how long people stay at locations from social media data to better understand mobility patterns.
- •For social media analysts: Enhance tools for inferring visit durations by integrating multimodal cues more effectively than current AI approaches.
Authors
Chenyang Yu, Zhaomin Xiao, Yan Huang, Eduardo Blanco
Abstract
Gardening in the backyard implies living there, while checking into an Airbnb more likely means staying for a few days. Such duration judgments are important for understanding human mobility, travel behavior, and social context. However, duration is rarely stated explicitly in social media posts and must instead be inferred from noisy textual, visual, and contextual cues. In this paper, we investigate whether vision-language models possess such duration commonsense. We create a new corpus of geotagged tweets co-annotated by humans and LLMs to serve as a testbed for this task. We find that state-of-the-art models struggle with duration inference, even when guided by multi-turn Chain-of-Thought prompting. Error analyses further reveal that models often fail to integrate critical contextual information, or they overly rely on prominent visual details while overlooking subtle hashtags and semantic nuances. These findings underscore the significant challenges remaining in context-aware multimodal understanding.