Viewpoint selection improves embodied question answering with 360 images
Informative Viewpoint Selection for Episodic-Memory Embodied Question Answering using Omnidirectional Images
Computer Vision and Pattern RecognitionRobotics
Summary
Answering questions about a place by looking around can be tricky when using 360-degree images because these images can be distorted and contain too much irrelevant information. The authors found a way to pick the most helpful smaller views from these big images based on the question being asked. This method keeps the important parts and reduces the extra clutter, helping the computer answer questions more accurately. They tested this on a home environment dataset and showed their way works better than previous ones, even when using much fewer views.
What this means in practice
- •For robot developers: Improve robots’ ability to answer questions about indoor environments using fewer, more relevant visual inputs from 360-degree cameras.
- •For virtual tour creators: Enhance interactive tours by selecting key views from panoramic images that best answer visitor questions about a location.
Authors
Kaname Kitamura, Asako Kanezaki
Abstract
Embodied Question Answering (EQA) requires agents to answer natural language questions about surrounding environments from visual observations. In this work, we focus on open-vocabulary episodic-memory EQA (EM-EQA), where an agent answers free-form questions using recorded observation histories. Omnidirectional images are promising for this task, as they provide wide field-of-view observations that can capture surrounding context without requiring explicit camera rotations. However, omnidirectional images introduce two challenges for EQA: (i) equirectangular projection causes severe geometric distortion that degrades vision-language model (VLM) recognition accuracy, and (ii) feeding equirectangular images directly into VLMs introduces excessive irrelevant background information, reducing answer accuracy and increasing the visual-token burden. To address these challenges, we propose a viewpoint selection method for EM-EQA using omnidirectional images. Our method converts equirectangular observations into perspective views via cubemap projection, estimates question-conditioned relevance with fine-tuned BLIP-2, and selects informative and diverse viewpoints through diversity-aware greedy selection. Experiments on the Habitat-Matterport 3D (HM3D) subset of OpenEQA show that our method achieves state-of-the-art model performance among the reported model results with equirectangular observations. Moreover, after removing rotation views, which reduces observation frames by 65.5%, our method largely maintains its answer accuracy.