Vision-language models control a toyota corolla in parking lot test
DrivingBench: Can Vision-Language Models Drive a Toyota Corolla?
RoboticsArtificial Intelligence
Summary
Driving a car is a complex skill that vision-language AI models have not been tested on before. The authors created DrivingBench, a test where these models control a real Toyota Corolla around cones in a parking lot at slow speeds. The models receive camera images and must steer and adjust speed directly, with delays included to simulate real conditions. Among the tested models, only one completed the course fully by the second try. The authors provide the tools and data openly to help others reproduce and improve this capability.
What this means in practice
- •For autonomous vehicle engineers: Test and improve vision-language model control of real cars under real-time constraints to enhance low-speed driving capabilities.
- •For robotics software developers: Use the DrivingBench platform and data to develop and validate integrated perception and control systems combining vision and language inputs for real robots.
Authors
Aditya Ramabadran, Simon Mahns, Tobias Gessler
Abstract
Frontier models excel at many digital benchmarks, yet their ability to drive a real car, an everyday human skill, remains largely untested. We present DrivingBench, to our knowledge the first benchmark where general-purpose vision-language models must drive a real car. Through three tools, the models see camera frames from a Toyota Corolla and directly command its steering and velocity around a parking lot cone course at low speeds. The car may continue moving while the model thinks and new commands replace the currently running one, so inference latency is part of the task, testing the models' abilities to observe, act, monitor, recover, and complete a long-horizon objective under such constraints. We benchmark GPT-6 Astra, Claude Fable 5.1, GPT-5.6 Sol, and Grok 4.6 in vendor-native harnesses (Codex, Claude Code, Cursor) with up to three attempts each in one conversation; Astra is the only model to finish the course, on its second attempt, with no other attempt passing 50% of the course. Two of the four models improved materially across attempts with retained context. We also detail the design principles behind our action interface, and show how the tool output format and the framing of the task combined to determine whether models would drive at all or refuse. We release our harness, prompts, course map, and traces with video and telemetry for reproducibility.