Vision language models learn to generate clearer references from gaze data
Learning to Refer from Estimated Listener Gaze
Computation and Language
Summary
This paper looks at how computer programs that describe images can get better at talking by watching where people look while listening. The researchers trained a model to predict human eye movements while hearing a description and used that to teach the program to say things more clearly and briefly. This approach helped the program make shorter descriptions that people understood more often. It shows a new way computers can improve communication by learning from how people process language, not just whether they understand it.
What this means in practice
- •For conversational ai developers: Improve virtual assistants to generate clearer and shorter descriptions based on modeling user gaze patterns during interaction.
- •For assistive technology designers: Design communication aids that adapt referring expressions by estimating listener attention from eye movement data to enhance clarity.
Authors
Téa Wright, Alane Suhr
Abstract
We propose to finetune vision-language models to generate more pragmatically optimal referring expressions by transforming observations of incremental listener comprehension, in the form of gaze scanpaths, into learning signals. During training, referring expressions are sampled from the speaker policy being optimized, conditioned on images and target referents; then, a neural listener estimating human gaze behavior maps from images and sampled referring expressions to scanpaths, each represented by a sequence of fixations, with each fixation corresponding to a word in the referring expression. We experiment with several approaches to convert fixation sequences and target referents into token- and sequence-level rewards, which are used to optimize policy parameters. Through evaluation with human listeners, we find that speaker policies trained with gaze-estimating listeners result in significantly more pragmatically-optimal references than base models, reducing sequence length from 15.4 down to 4.0 words while increasing referential success from 75.2 up to 80.0%. Our work demonstrates a promising opportunity for learning to generate utterances through language-based interaction, not only from the explicit signal of communicative success, but also from implicitly-available observations of a listener's process of comprehension.