Summary
People who are blind or have low vision face many challenges doing everyday tasks, so AI helpers that can guide them would be useful. The authors explain that current AI tests mostly use pictures and text and don’t try other kinds of information like gaze or motion, which are available from wearable devices. They created VISTA, a new collection of videos and sensor data showing tasks from a first-person view, to test AI on guiding blind or low-vision users toward their goals. They tested some AI models and found these are good at describing scenes but still not great at giving useful step-by-step help. VISTA hopes to push AI to better assist people with vision loss in daily life.
What this means in practice
- •For assistive technology developers: Develop AI systems that combine visual, gaze, and movement data to better guide blind and low-vision users through daily tasks using the VISTA benchmark.$Commercial implications: It enables creation of advanced wearable assistance products specifically designed for visually impaired users by providing a unique multimodal dataset for training and evaluation.
- •For robotics system integrators: Integrate multimodal sensors into wearable robots to improve environment understanding and goal-directed assistance using insights from the VISTA dataset.
Authors
Noé Mosca, Mohamed Rayan Barhdadi, Muhammad Hamza, Joshua Yeh, Suryodaya Bikram Shahi, Zhiyan Li, Selim-Antoine Lali, Weiyue Li, Hasan Kurban, Erchin Serpedin, Gang Luo, Yue Zhang, Minghan Li, Mengyu Wang
Abstract
Blindness and low vision limit many people's ability to complete daily tasks, motivating AI assistants that provide useful guidance. Yet, existing benchmarks do not adequately test whether vision models can provide effective everyday assistance. In particular, i) existing benchmarks primarily rely on visual and textual inputs, with limited support for other modalities that are increasingly available in egocentric wearable devices such as gaze, inertial measurements, spatial trajectories, and others; and ii) existing evaluations emphasize scene understanding and question answering, with limited coverage of goal-conditioned guidance for blind and low-vision users across everyday tasks. To bridge this gap, we introduce VISTA (Visually Impaired Scene and Task Annotations), a multimodal egocentric benchmark for goal-oriented assistance for blind and low-vision users. VISTA contains $\sim$1,000 egocentric recordings captured with Project Aria glasses; each recording provides synchronized sensor streams, six of which serve as model inputs, and three different types of annotation, covering ten task categories (TC), from object localization to hazard detection. We evaluate 14 representative vision-language models zero-shot on VISTA and design a lightweight multimodal model baseline that integrates all six modalities. Experimental results show that, although existing models can often describe egocentric scenes, they remain limited in generating goal-directed assistive guidance. VISTA thus provides a benchmark for evaluating multimodal models that could improve everyday accessibility.