Vision language action agent improves driving with language based memory
Vision-Language-Action Autonomous Driving Agent with Language-based Memory
Computer Vision and Pattern RecognitionRobotics
Summary
Self-driving cars have trouble remembering long sequences of events needed for tricky situations like all-way stops. The authors created a new driving agent, AD-Memo, that uses language to store memories of important objects and scenes by extending its reasoning process. This memory lets the agent better understand complex driving scenarios and answer questions about what it sees. Training used supervised learning plus a new reinforcement learning method that improves the agent's decision-making over time.
What this means in practice
- •For autonomous vehicle engineers: Enhance autonomous driving systems with interpretable language-based memory to handle complex driving tasks over longer timeframes.
- •For robotics platform developers: Integrate plug-and-play language memory modules from AD-Memo into robot control systems for improved scene understanding and decision making.
Authors
Kai Yan, Xiangyu Chen, Yulong Cao, Alex Naumann, Peter Karkus, Yan Wang, Jef Packer, Alex Schwing, Yuxiong Wang, Boris Ivanovic, Wenjie Luo, Marco Pavone
Abstract
Vision-Language-Action (VLA) foundation models have recently emerged as one of the prevailing solutions for autonomous driving, as they can utilize knowledge acquired during vision-language pretraining for accurate and interpretable driving. However, VLAs can take only a limited number of frames as visual input due to the high token cost of an image, which is problematic for memory-dependent tasks such as determining the arrival order at all-way stops and long-horizon driving scene understanding. Existing solutions use latent vector memories accessed through cross-attention, which are neither interpretable nor portable. In this paper, we propose AD-Memo, a general-purpose VLA driving agent with language-based memory. The agent outputs memory as an extension of its Chain-of-Thought (CoT) to record surrounding objects critical to driving; this memory becomes part of the agent's future input. We curate memory-based datasets and train VLAs with a two-stage recipe: Supervised Fine-Tuning (SFT) and \textit{Da Capo}, a novel semi-closed-loop Reinforcement Learning (RL) algorithm which uses trajectory-level advantage for memory and step-level advantage for driving, leading to better credit assignment. Across scenarios such as all-way stops and general driving, AD-Memo improves driving quality, enables better question answering on driving scenes, and produces plug-and-play memory for other models.