Summary
Predicting where people look in images is useful for design and interaction but existing methods only give vague heat maps. The authors developed OpenVAM, a system that not only shows where attention peaks occur but also explains what objects these peaks relate to and why they attract attention. It works on different kinds of images, like natural scenes and web pages, and stays reliable when shifting between types of images. OpenVAM uses a two-part design: one part focuses on locating attention precisely, while the other generates language-based explanations connected to those spots. The team trained it in stages to keep accuracy and add meaningful explanations without disrupting the core attention prediction.
What this means in practice
- •For ui designers: Improve web or app designs by automatically linking user attention points to specific interface elements with natural language explanations.
- •For roboticists: Enable robots to predict human gaze locations and understand why people focus certain scene elements, improving interaction and task planning.
Authors
Kiana Hooshanfar, Amirhossein Kazerouni, Alireza Hosseini, Michael Brudno, Babak Taati
Abstract
Predicting human gaze is a core capability for applications ranging from web/UI design analysis to robotics and human-computer interaction. Yet, most visual attention modeling methods output only a dense saliency map, which is often insufficient for action: practitioners need to connect attention peaks to discrete elements in the scene (what) and understand the drivers of those peaks in context (why), while remaining robust to domain shift across natural images, commercial content, and UI/web layouts. We, therefore, introduce OpenVAM (Open-world Visual Attention Modeling with VLMs), a unified framework that jointly addresses universality and explainability across heterogeneous domains (natural scenes, commercial imagery, and UI/web layouts) and supervision modalities. OpenVAM adopts a decoupled-but-aligned design: a dedicated dense visual pathway provides stable, spatially precise localization, while an instruction-following vision--language semantic head generates grounded what/why explanations conditioned on the same image and data-type prompts. A three-stage training strategy preserves strong localization priors while progressively introducing language grounding and improving explanation alignment via parameter-efficient adaptation without perturbing the saliency branch. We further propose a scalable pipeline to generate multi-domain saliency-reason annotations for training and systematic evaluation. Experiments across diverse datasets show that OpenVAM improves robustness under domain shift while producing image-grounded explanations that make saliency predictions more interpretable.