Selective neuro symbolic reasoning boosts autonomous driving question answering
From Scene Graphs to Answers: Selective Neuro-Symbolic Reasoning for Autonomous Driving
Artificial Intelligence
Summary
Understanding questions about a self-driving car’s surroundings needs careful thinking about objects and their relationships over time. The authors propose a new system that first tries to answer questions by exact reasoning on a detailed map of objects and events. If that fails, it then uses a language model for more flexible interpretation, giving it only the relevant information it needs. This approach improves accuracy and efficiency on a real dataset compared to just using language models alone.
What this means in practice
- •For autonomous vehicle developers: Improve understanding and answering of questions about complex driving scenes by combining exact symbolic operations with language models for better accuracy and efficiency.
- •For robotics software teams: Enhance robot reasoning by using hierarchical scene graphs and selective symbolic execution before semantic interpretation to handle spatial and temporal queries more reliably.
Authors
Yiyao Wang, Pei Liu, Fangzhou Liu, Jun Ma
Abstract
Autonomous-driving question answering requires reasoning over structured scene information, yet existing vision-language approaches largely delegate heterogeneous reasoning operations to a single neural inference process. We argue that this uniform strategy overlooks a fundamental distinction: some queries admit exact symbolic solutions, while others require semantic interpretation. We introduce a query-adaptive neuro-symbolic reasoning framework that explicitly allocates computation according to the nature of the query. At its core is a hierarchical Spatiotemporal Scene Graph (STSG) that separates persistent object identities from frame-specific states and represents spatial relations and temporal transitions as explicit directed structures. Given a query, a symbolic executor first attempts to resolve it through exact graph operations; only when symbolic execution abstains is an LLM invoked for semantic reasoning. For these unresolved queries, query-conditioned graph retrieval and evidence filtering preserve relation direction, temporal locality, and object semantics, providing the LLM with compact and verified task-relevant evidence. This design shifts the role of the LLM from a universal reasoning engine to a targeted semantic reasoner, while allowing deterministic computation to be handled exactly and efficiently. We evaluate the framework on 5,916 NuScenes-QA questions across all ten scenes of nuScenes v1.0-mini under an oracle-perception setting. The complete system achieves 80.63 percent overall accuracy with GPT-5.4-mini, improving over the corresponding LLM-only configuration by 5.48 percentage points; with DeepSeek-V4-Flash, the improvement reaches 6.64 points. The largest gains occur on counting questions, with improvements of 10.20 and 12.61 points, respectively. These results show that selective reasoning improves both accuracy and inference efficiency.