FIVE-VLA speeds up autonomous driving with memory and fewer tokens
FIVE-VLA: Fast and EffectIVE Autonomous Driving with Recurrent Action Memory
Computer Vision and Pattern RecognitionRobotics
Summary
Autonomous driving systems often struggle with slow processing and remembering recent actions. The researchers created FIVE-VLA, which processes high-resolution images more efficiently by using fewer data tokens and avoids some complicated steps. It also includes a memory feature that helps the system remember past actions to make better driving decisions like overtaking safely. This new approach is faster, uses fewer computer resources, and drives more safely in tests than older methods.
What this means in practice
- •For autonomous vehicle engineers: Build faster and safer driving systems that better remember past actions to improve maneuver performance on complex routes.
- •For robotics software developers: Integrate efficient vision encoders and action memory modules to speed up real-time decision-making in robots navigating dynamic environments.
Authors
Kemal Oksuz, Alexandru Buburuzan, Yuhan Yao, Puneet K. Dokania
Abstract
State-of-the-art vision-language-action models (VLA) for autonomous driving face critical limitations: excessive parameter counts, inefficient high-resolution image processing, and lack of temporal memory. We introduce Fast and EffectIVE VLA (FIVE-VLA) to address these through two key contributions. First, we employ an efficient vision encoder that processes high-resolution ($448 \times 896$) images while generating only 98 tokens, over $5\times$ fewer than existing approaches, and bypass text generation entirely for single-pass trajectory prediction. Second, we propose Recurrent Action Memory (RAM), a lightweight module that conditions action prediction on previous action tokens, providing temporal context critical for manoeuvres such as overtaking and emergency braking. With only 641M parameters, FIVE-VLA completes $\sim$10% more routes without traffic rule infractions than the previous state-of-the-art VLA on the challenging Bench2Drive closed-loop driving benchmark. Non-reactive open-loop simulation on the large-scale real-world NVIDIA Physical AI AV dataset shows 10.2% and 7.7% lower collision-violation rates than SimLingo in single- and four-view settings, respectively. Additionally, FIVE-VLA runs at $\sim$30 fps on an A100 and $\sim$4 fps on a T4 GPU (proxy to an edge device), representing an 8-30$\times$ speedup over previous methods.