Rethinking Language's Role in Efficient VLA for Autonomous Vehicles: Toward Smarter, Trustworthy Driving
2026-08-31 • Robotics
Robotics
AI summaryⓘ
The authors discuss how combining language understanding with vision and action helps self-driving cars make smarter decisions. They point out that using language during driving is costly because it requires extra time and memory, which are limited in cars. To tackle this, they categorize different ways language can be used at the time of driving, from only using it during training to generating language every moment. They compare these methods on common driving tests and look at how language tools from other fields are adapted for autonomous driving, highlighting practical challenges. They also plan to keep sharing updated resources on this topic.
Vision-Language-ActionAutonomous DrivingPerceptionLanguage InferenceLatencyMemory BudgetAutoregressive DecodingBenchmarkingNatural Language ProcessingLanguage Residue Taxonomy
Authors
Tongfei Guo, Lili Su
Abstract
Vision-Language-Action (VLA) models are reshaping autonomous driving (AD) by unifying perception, reasoning, and control through language, enabling semantic grounding, interpretable decisions, and better long-tail generalization. But language is expensive onboard: latency and memory budgets are tight, and autoregressive decoding is inherently sequential. This work reframes the central question as when and where language should act at inference, since inference cost recurs at every deployed frame while training cost is paid once. We introduce the Language Residue taxonomy to organize methods by their inference-time use of language: train-time-only supervision (L1), latent non-textual reasoning (L2), conditional invocation (L3), and full per-frame generation (L4). We review representative methods and tag each across five deployment axes (latency, parameters, memory, FLOPs, tokens), analyzing them on major open- and closed-loop driving benchmarks (e.g., nuScenes, NAVSIM, Bench2Drive). We further trace how efficient methods from NLP/LLM are adapted in AD, identifying the constraints and motivations driving these adaptations. A continuously updated repository will be available at Github.