Causal reasoning is limited in vision language autonomous driving models
CausalDriveBench: Evaluating Causal Reasoning in Vision-Language-Action Models for Autonomous Driving
Computer Vision and Pattern RecognitionRobotics
Summary
The authors study how well AI models for self-driving cars understand cause and effect in driving scenes. They create a new test called CausalDriveBench that asks questions and looks at alternative driving paths to check if models really grasp what causes what. Their tests show many models don’t understand causal relationships well, even if they give good answers or drive safely in normal scenes. This means current self-driving AI might not truly think about cause and effect in complex situations.
What this means in practice
- •For autonomous vehicle engineers: Evaluate and improve self-driving AI systems’ ability to understand cause-effect in complex road scenes using the CausalDriveBench framework.
- •For ai system testers: Use causal question answering and counterfactual trajectory generation to better assess AI models beyond surface-level accuracy.
Authors
Narendiran Chembu, Navvrat Rao, Shreedhar Shreeshail Kodate, Gayatri Srujana Banda, Arko Sarkar, Abhinav Khanna, Rajarshee Das, Umesh Kanala, Siddarth Khandelwal, Kumar Aman, Aish Dubey, Kaustubh Beedkar, Arjun Jain
Abstract
Vision-Language-Action (VLA) models for autonomous driving produce natural-language reasoning alongside predicted trajectories, but whether this reasoning reflects the causal structure of the scene remains untested. We introduce CausalDriveBench, an evaluation framework grounded in Pearl's Causal Hierarchy (PCH) that tests causal reasoning in driving-specific VLAs through structured visual question answering (QA) and alternative-trajectory prediction. To this end, we construct causal scene graphs over nuScenes that distinguish causally active, dormant, and distractor entities, separating perceptual salience from causal relevance. The benchmark spans all four rungs of PCH (association, intervention, and counterfactual along with causal discovery) for QA generation. For the higher rungs, we additionally provide reference trajectories under specified scene modifications, enabling action-level verification that complements reasoning-level evaluation. In total, the benchmark contains 7,285 verified causal QA pairs and 1,000 counterfactual trajectories derived from nuScenes. We evaluate 10 driving-specific VLAs and 3 general-purpose VLMs, and report three findings. First, the best model reaches only 70.6% QA accuracy, and 4 of 13 models score below random chance. Second, comparing each driving VLA to the general-purpose VLM that shares its language backbone, the cost of driving fine-tuning ranges from 2 to 34 percentage points on causal QA, with post-training design explaining the spread. Third, causal QA and trajectory accuracy are statistically uncorrelated across models: under counterfactual prompts, predicted trajectories either over-react or collapse onto the observed-scene baseline. Taken together, these results show that neither fluent rationales nor accurate observed-scene trajectories constitute evidence of causal understanding.