Vision language driving models trained with fewer instructions match baselines
Less Language, More Latents: Annotation-Efficient VLAs for Driving
RoboticsMachine Learning
Summary
Teaching self-driving cars to understand natural language instructions is hard because it takes a lot of labeled examples. The authors found a way to use mostly unlabeled driving videos and actions to first create a simpler set of driving intentions. Then, they use a small amount of language-labeled examples to link language to these simplified driving codes. Finally, they train a model to drive using the full dataset with fewer language labels, achieving performance as good as fully labeled approaches.
What this means in practice
- •For autonomous vehicle engineers: Train driving models that understand human language instructions using far fewer manually labeled driving videos.
- •For robotics software developers: Develop robot control systems that connect language commands to high-level actions with minimal language annotation effort.
Authors
Alexey Zakharov, Kemal Oksuz, Puneet K. Dokania
Abstract
Vision-language-action models (VLA) promise human-steerable autonomous driving, but their training is bottlenecked by the scarcity of frames paired with natural-language instructions: while camera streams and expert trajectories are logged at scale, language annotations (e.g., turn left at the intersection) remain scarce and expensive to acquire. To address this challenge, we introduce Latent Action Driving Annotations (LADA), a three-stage pipeline that transforms abundant unlabelled observation-trajectory pairs into a substrate for language-conditioned control. First, we train a latent action model with a vector-quantised bottleneck, producing a compact codebook of high-level vehicle intents. Second, a small language-annotated subset is used to train a vision-language translator to map observations and language instructions into this codebook. Third, we train a driving VLA on observation-latent-action pairs over the full unlabelled corpus. Using fewer than 5% of language annotations and without leveraging any auxiliary chain-of-thought reasoning or visual question answering streams, LADA achieves a Driving Score of 87.98 and a Success Rate of 70.46% on the closed-loop Bench2Drive benchmark, matching or surpassing fully supervised baselines.