Human annotated dataset helps test reasoning in self driving cars
CASCADE: A Spatio-Temporal-Causal Reasoning Representation and Dataset for Driving
Computer Vision and Pattern Recognition
Summary
Self-driving cars need to understand how different things on the road affect each other to make smart decisions. The authors created CASCADE, a new way to describe driving scenes that shows what each driver or object is doing, where, and how their actions influence others over time. They also made a large, carefully checked dataset using this description for over 2,000 driving clips. This dataset helps check if self-driving models really understand these connections instead of guessing. It can also be used to compare and improve other systems that try to explain driving behavior.
autonomous drivingcausal reasoningspatio-temporal representationscene graphdataset annotationego vehicleactor actionsmachine verificationPhysicalAI datasethuman annotation
Authors
Jenny Schmalfuss, Despoina Paschalidou, Simon Gerstenecker, German Ros, Jose M. Alvarez
Abstract
Reasoning is a promising route to the generalization that autonomous driving requires in the long tail, as it can infer how the elements of a scene depend on one another and traverse those dependencies to conclusions beyond what is observed. Yet it is hard to tell whether a model's conclusions follow the scene's dependencies, because no driving representation makes them explicit enough to test against. Text-based reasoning traces lack spatio-temporal grounding, spatio-temporal scene graphs lack causal links, and reasoning annotations at scale are increasingly model-generated and hard to verify. To this end, we introduce CASCADE (Causal Spatio-Temporal Analysis of Driving Environments), which encompasses two components: (1) a structured scene representation for reasoning in driving scenes and (2) a human-annotated dataset built on it. For every actor that interacts with the ego vehicle, the CASCADE representation records frame-by-frame, for as long as the actor is visible, what action is taken, where it occurs, and how it depends on the actions and states of others. The resulting structure makes reasoning predictions machine-verifiable: they can be scored against it element by element, without relying on (M)LLM judges. The CASCADE dataset provides comprehensive human annotations for 2,066 driving clips of the PhysicalAI dataset, with over 34K elements that establish the spatio-temporal and causal context of each scene, including 8.6K time-stamped ego and agent actions, 3.7K causal links and 2.9K potential influences, and 6.1K annotations for agents, objects, traffic lights, and environments. Being entirely human-annotated, CASCADE provides the reference for this comparison: benchmarking the reasoning abilities of Physical AI models, and verifying the quality of automatically generated reasoning labels. The CASCADE dataset is available at https://huggingface.co/datasets/nvidia/cascade.