Neuro-symbolic captions bring clear scene maps to autonomous driving
NeuroSymbEAD: A Large Scale Neuro-Symbolic Caption Dataset for Omni-Directional Embodied Autonomous Driving
Computer Vision and Pattern Recognition
Summary
Understanding complex driving scenes is difficult for autonomous vehicles because they need to recognize many objects and their relationships. This paper presents NeuroSymbEAD, a dataset that combines visual data with structured information like object types, positions, and directions, all described in clear, multi-level text captions. The authors use this dataset to help computers better explain and reason about the driving environment from the vehicle's point of view. This approach aims to improve how autonomous cars understand traffic scenes and make safer decisions.
What this means in practice
- •For autonomous vehicle engineers: Generate structured text descriptions of driving scenes to improve vehicle perception and decision-making accuracy in real-world environments.
- •For robotics developers: Create interpretable language-based representations of 3D surroundings to enhance robot navigation and situational understanding.
Tested on one dataset.
Authors
Muhammad Ahmed Ullah Khan, Mohammed Elamine, Sheikh Talha Uddin, Didier Stricker, Sk Aziz Ali, Muhammad Zeshan Afzal
Abstract
This paper introduces NeuroSymbEAD, a large-scale neuro-symbolic caption dataset featuring an ego-centric knowledge graph (KG) of static and dynamic objects annotated with classes, categories, heading directions, orientations, and distances from the ego-vehicle. These annotations are used on the KITTI-360 dataset to generate multilevel textual captions representing a lightweight version of an ego-centric scene map. Outdoor scene-map reconstruction, visual recognition, and object grounding establish baselines for driving common sense and traffic/scene understanding. For these purposes, natural language-based grounded captioning of objects and their complex relationships is a widely adopted contextual representation for indoor scene tasks. Neuro-symbolic representations have proven effective in handling structured information for various computer vision and language applications. Our data annotation pipeline allows the generation of varied map segments, populating simulated or real objects within the bounding boxes predicted by any 3D object detection network, and building hierarchical text captions. We benchmark our neuro-symbolic and ontological caption generation using pre-trained grounding and learned auto-regressive captioning networks. By converting 3D driving scenes into structured ego-centric language, NeuroSymbEAD provides a benchmark for vision-language and foundation models for traffic-scene explanation, 3D reasoning, and interpretable autonomous-driving perception.