Visual question answering improves consistency for road traffic scenes
Answer with Evidence: Consistency-Aware Grounded Visual Question Answering for Roadside Traffic Scenes
Computer Vision and Pattern RecognitionHuman-Computer Interaction
Summary
Answering questions about roadside traffic scenes is hard because the answers need to match specific parts of images exactly. The authors found that current systems often give answers that don’t line up correctly with the places they point to in pictures. They created a new large dataset and a way to check if answers and evidence match better. They also made a system that picks out relevant image parts first, then answers, improving how well answers and images agree.
What this means in practice
- •For autonomous vehicle teams: Improve roadside scene understanding by ensuring answers to visual traffic questions correctly match specific image regions, enhancing safety-critical decision making.
- •For surveillance system developers: Generate consistent and evidence-backed answers to questions about traffic environments from video feeds, aiding situational awareness.
Authors
Runwei Guan, Rongsheng Hu, Shangshu Chen, Ningwei Ouyang, Shaofeng Liang, Heyi Lin, Jinjing Zhu, Yang Shi, Dongming Wu, Daizong Liu, Henghui Ding, Hui Xiong
Abstract
Roadside traffic reasoning requires every free-form textual claim to be backed by visual evidence. Existing grounded multimodal large language models (MLLMs) frequently exhibit say-point mismatch, in which the textual answer contradicts the bounding boxes the model localizes. Evaluation metrics that score answers and boxes separately leave this failure unpenalized. We trace the mismatch to the conventional answer-then-ground factorization, which commits to a numerical claim before any object is enumerated. To measure it, we build RoadSceneVQA-G, a benchmark of 34.7K question-answer pairs in which every free-form answer is linked to the set of boxes that witnesses it, and we propose the Answer-Grounding Consistency (AGC) evaluation suite. To address it, we introduce Enumerate-then-Answer (EtA), which reverses the generation order so that answer-evidence agreement becomes a property of the output structure, and Enumeration-Consistent Policy Optimization (ECPO), a reinforcement learning stage that uses the union of multiple rollouts as a recall teacher without ground-truth boxes. EtA raises say-point consistency from 26.6\% to 93.7\% and grounding F1 from 52.2\% to 73.0\%, and ECPO further increases F1 to 75.6\% without per-box supervision. On gRefCOCO, the same framework outperforms the strongest compared method, indicating that it transfers beyond traffic scenes. The project is available at \url{https://github.com/GuanRunwei/RoadSceneVQA-G}.