Decoupled semantic analysis improves traffic scene question answering and descriptions
Sim-to-Real Traffic Scene Understanding by Decoupling Semantics from Caption Generation with V-JEPA
Computer Vision and Pattern Recognition
Summary
Understanding traffic scenes from videos is hard, especially when moving from computer-generated images to real ones. The authors separate the task into two steps: first, they answer specific questions by extracting clear facts about the scene, then they use these facts to write descriptions. By fixing mistakes in the facts before generating text, their method makes fewer errors and gives better answers and descriptions. Their approach performed best in a major competition focused on traffic scene understanding.
What this means in practice
- •For traffic monitoring teams: Produce more accurate answers to traffic-related visual questions and descriptions despite changes from synthetic to real video data.
- •For autonomous vehicle developers: Improve understanding of pedestrian and vehicle behaviors via structured semantic reasoning to support safer on-road decision making.
Authors
Nguyen Hoai Thuong Bui, Thanh Nguyen Vo, Trinh Tra Giang Nguyen, Ha Duc Bui
Abstract
Track 2 of the AI City Challenge 2026 requires both visual question answering (VQA) and traffic event description generation under a challenging synthetic-to real domain shift. Existing vision-language approaches often entangle semantic understanding with language generation, making them susceptible to hallucination and inconsistent reasoning across event phases. In this work, we propose a decoupled semantic understanding framework that first resolves predefined traffic questions into structured semantic facts and subsequently uses these facts to guide caption generation. A frozen V-JEPA encoder extracts predictive scene representations, while a lightweight Llama-based predictor produces answers for VQA queries. To improve reliability, we introduce a training-free structured refinement mechanism that exploits statistical priors, inter-question relationships, and temporal event consistency to correct prediction errors. The refined semantic facts are then provided to Qwen3-VL-8B to generate pedestrian and vehicle descriptions for each traffic event. Experimental results on the official 2026 AI City Challenge Track 2 benchmark show that the proposed method achieves 87.09% VQA accuracy and an overall S2 score of 60.0853, ranking first among all participating teams. These results demonstrate that predictive world representations combined with structured semantic refinement enable more accurate and reliable traffic understanding, leading to higher-quality lan guage generation.