LLMs tested on reasoning with uncertain visual evidence for questions
How Well Do LLMs Reason with Noisy Evidence? An Active Visual Reasoning Benchmark
Artificial Intelligence
Summary
When answering questions using images, computers often have to deal with unclear or noisy information. The authors created a new challenge called VisualNoiseQA where a language AI asks a visual AI multiple times to get clues before answering questions. This tests how well the language AI can handle uncertain visual hints and decide when it has enough information to answer. The study helps understand how these AIs work together when their visual inputs are not perfect.
What this means in practice
- •For robot builders: Design robots that ask multiple sensory questions to improve understanding in uncertain environments by measuring sensor uncertainty.
- •For business intelligence teams: Integrate systems that assess uncertainty in visual data when extracting insights from charts and images for more reliable decision-making.
Authors
Bach Nguyen, Zhaonan Li, Mau Son Nguyen, Sanika Chavan, Nilay Kumar, Hong Anh Nguyen, Khoa Vo, Ben Zhou
Abstract
Real-world reasoning rarely reduces to static question answering: agents must actively gather information from tools and sensors that are often noisy and unreliable. Yet most existing active reasoning benchmarks assume that environmental feedback is trustworthy, or introduce noise without exposing an explicit, calibrated uncertainty signal, leaving open how LLMs should reason when the evidence itself is uncertain. We introduce VisualNoiseQA, a novel benchmark for active reasoning under noisy visual feedback. A text-only LLM must solve VQA problems by iteratively querying a fixed, off-the-shelf VLM treated as a stochastic visual sensor. For each query, we draw multiple samples and expose an empirical uncertainty signal via self-consistency, enabling the reasoner to probe from different angles and decide what to ask next and when to stop. Our construction is automatic and scalable: starting from diverse VQA sources and two noisy VLMs, we retain only questions where the sensor is inconsistent yet human-solvable. We evaluate multiple LLM reasoners on 1,000 instances spanning perception, chart understanding, and knowledge-intensive reasoning. VisualNoiseQA thus provides a controlled playground to study how different LLMs exploit uncertainty signals for robust reasoning.