Vision language models communicate visually but differ from humans

A Heart Means Recursion: Exploring Visual Meaning and Communication in VLMs

Artificial IntelligenceComputation and LanguageComputer Vision and Pattern Recognition

Summary

Vision-language models (VLMs) can use simple pictures like red hearts to influence how they solve problems, even in surprising ways like encouraging repeated steps in coding tasks. The researchers found that while models send visual signals that seem easy for humans to understand, other models don't always interpret them the same way, creating a kind of communication gap. Sometimes these models come up with creative ways to share ideas visually with each other, but some messages are confusing to people. This work shows that VLMs have potential for visual communication but do it differently than humans do.

What this means in practice

  • For ai developers: Design visually guided AI systems that use simple symbols to influence model problem-solving strategies in new ways.
  • For interactive media designers: Create AI-driven tools that generate visual cues to communicate abstract ideas between models for storytelling or design tasks.

Authors

Alexa R. Tartaglini, Zhengxuan Wu, Thaddäus Wiedemer, Martin Wattenberg, Been Kim, Yuxuan Li

Abstract

Do vision-language models (VLMs) share our ability to communicate visually? We share preliminary findings, starting from a surprising observation: simple visual symbols like a red heart can strongly influence model behavior on seemingly unrelated tasks, such as steering models towards recursive solutions on simple coding problems. Notably, this steering effect is largely implicit, as models' direct interpretations and thinking traces about visual symbols diverge substantially from their observed behavior. In visual communication between models, this leads to a generation--interpretation gap: while models tend to send shape primitives that they (and humans) find intuitive, these visual cues often fail to steer other models. In further communication games involving abstract concepts and stories, models can creatively use visuals to transmit meaning to one another, although some communications remain non-intuitive to humans. These findings reveal that VLMs might be capable of rich, creative, and visually-grounded communication, but their interpretation and production of visual meaning diverges from human conventions in surprising ways. Together this work highlights both the promise and challenges of visual communication between humans and VLMs.