RoboVista: Evaluating Vision Language Models for Diverse Robot Applications
2026-07-06 • Robotics
Robotics
AI summaryⓘ
The authors introduce Robot Question Answering (RQA), a new way to test how well Vision-Language Models (VLMs) can help robots make decisions by breaking down tasks into smaller parts. They created RoboVista, a benchmark with 474 questions from different types of robots used in fields like farming, industry, and surgery, each with expert explanations. Their tests show current VLMs still struggle with many robot tasks. They also found that performance on RoboVista relates well to how robots do in real life.
Vision-Language Models (VLMs)Robot Question Answering (RQA)benchmarkVisual Question Answering (VQA)robotic reasoningrobot tasksmodular evaluationRoboVistarobot embodimentsreal-world robotics
Authors
Shuangyu Xie, Kaiyuan Chen, Ziyang Chen, Simeon Adebola, Yixuan Huang, Zehan Ma, Tianshuang Qiu, Wentao Yuan, Dhruv Shah, Pannag R. Sanketi, Ken Goldberg
Abstract
Diverse applications for robotics, such as industry and agriculture, require robots to operate across various embodiments, changing visual conditions, and complex planning. Vision-Language Models (VLMs) offer a promising foundation for general-purpose and interpretable robotic reasoning. Aligning VLMs with diverse robot applications requires a modular understanding of the individual decision components that underlie robotic behavior. Capturing such structure is challenging for conventional robot benchmarks that are primarily based on teleoperated, end-to-end datasets. We propose Robot Question Answering (RQA), a modular evaluation framework and RoboVista, a benchmark curated from real robotic systems, research papers, and expert annotations. RoboVista contains 474 Visual Question Answering (VQA) instances with human annotated reasoning and covers 39 unique task types in agricultural, industrial, domestic, surgical robotics, autonomous driving, and open robot datasets. Experiments on RoboVista show that state-of-the-art VLMs exhibit substantial gaps. Physical robot experiments suggest strong correlation between RoboVista performance and real-world task execution.