Embodied agent benchmark tests active visual reasoning in real scenes
JRDB-AVR: An Active Visual Reasoning Benchmark for Embodied Agents in Real-World Environments
Artificial IntelligenceComputer Vision and Pattern RecognitionRobotics
Summary
Sometimes robots or AI agents need to look around actively to find the right clues to answer questions about their surroundings. The authors created a new test called JRDB-AVR that checks if these agents can ask to see certain views or moments before giving an answer. They found many AI systems give correct answers without actually seeing the evidence, which means they guess or use shortcuts. Their benchmark helps measure both the correctness of answers and whether agents truly see and use the right information.
What this means in practice
- •For robotics developers: Evaluate and improve robots that must actively explore real environments to answer questions requiring multi-step visual reasoning.
- •For augmented reality designers: Develop AR systems that ensure visual answers are based on verified views and moments, improving trust in spatial understanding.
Authors
Zhixi Cai, Fucai Ke, Sukai Huang, Maria Garcia de la Banda, Peter J. Stuckey, Gholamreza Haffari, Hamid Rezatofighi
Abstract
In complex embodied visual reasoning scenarios, an agent often has only a limited field of view, and the evidence needed to answer a question may be distributed across time, viewpoint, and interacting objects. A model may therefore give a plausible answer without ever observing the relevant object, time, or view that supports it. Current visual reasoning benchmarks largely evaluate passive observations and final answers, overlooking settings that require active reasoning and evidence acquisition. We introduce JRDB-AVR, a benchmark derived from existing real-world JRDB robotics data through a structured question-generation engine that turns this gap into an explicit evaluation: an embodied agentic system receives a visual reasoning question, requests bounded observations by timestamp and viewing angle, and is evaluated on both the final answer and the grounded visual evidence supporting it. The benchmark contains diverse questions over multiple real-world environments involving temporal search, viewpoint selection, and human-oriented compositional reasoning. We also introduce JRDB-AVR-Agent, a reference active reasoning agentic method that maintains an explicit observation-grounded graph-based world model and answers through solving. Experiments reveal a substantial gap between answer accuracy and evidence accuracy in current baselines, showing that current VLMs can produce unsupported correct answers and that active evidence-aware evaluation is necessary for embodied visual reasoning. Code and benchmark are available at https://github.com/ControlNet/JRDB-AVR.