GRAVA improves autonomous driving by grounding language in scenes and actions
GRAVA: Grounded Reasoning-to-Action Representation and Learning for Autonomous Driving
Computer Vision and Pattern RecognitionRobotics
Summary
Driving AI needs to understand what it sees and say why before it decides how to act. The authors created GRAVA, a method that helps the AI link words about the scene to real things it can see and touch, then plan its driving. GRAVA also learns from lots of examples to get better at this reasoning, which leads to safer and smarter driving decisions. This method worked better than previous AI models on standard driving tests.
What this means in practice
- •For autonomous vehicle engineers: Develop driving systems that better connect visual scene understanding to safe driving actions using grounded reasoning techniques.
- •For robotics software developers: Build embodied agents that reason about their environments and generate actionable plans grounded in both language and perception.
Authors
Xiao Liu, Haoyu Li, Jianghao Leng, Lin Wang, Chao Sun
Abstract
Driving vision-language-action (VLA) models increasingly reason before acting, but their intermediate reasoning is often weakly grounded in physical scene evidence and loosely connected to executable behavior. We present GRAVA, a framework built around Grounded Reasoning-to-Action (GRA), which unifies grounding, reasoning, and action generation in a single autoregressive stream. GRA links action-relevant language references to 2D visual regions and ego-centric physical states, organizes object interactions and decisions in a trajectory-anchored typed graph, and serializes this structure into grounded reasoning. A single VLM generates this reasoning followed by a compact Executable Planner action that is deterministically decoded into a continuous trajectory. We further introduce an agentic GRA data construction pipeline that combines forward scene grounding with backward trajectory anchoring, and use it to build GR-NavSim with 2.2M grounded question-answer pairs and 70K GRA reasoning traces. A progressive training strategy develops grounded cognition through pre-training, establishes the reasoning-to-action interface through imitation, and improves driving behavior through reinforcement learning and exploration. Using about 60% of the available human driving demonstrations for action supervision, GRAVA-8B achieves state-of-the-art performance among purely autoregressive driving models on the full NAVSIM benchmark. On an internal long-tail benchmark, full GRA improves key-object compliance and Closed-loop Driving Score by 19.3% and 20.5% over action-only prediction, respectively. These results show the benefit of preserving action-relevant physical evidence from grounded reasoning through executable action generation.