Vision language models latency cuts robotic task success with moving objects
VLALagBench: Understanding When and Why VLA Inference Latency Matters
Robotics
Summary
Robots that see, understand, and act rely on quick thinking to catch or move objects smoothly. The authors show that a delay as short as 100 milliseconds in the robot’s thinking can greatly reduce success in tasks involving moving objects. They created a benchmark called VLALagBench to measure how timing affects robot tasks and found that running thinking and acting at the same time can help. They also built a model to predict success rates based on delay, helping choose the best setup for each robot system.
What this means in practice
- •For robotics engineers: Select robot vision-language model configurations that maximize success rates under real-world latency constraints using the VLALagBench benchmark and predictive model.
- •For autonomous vehicle developers: Estimate and optimize perception-to-action delays in vehicle systems to improve response accuracy for moving obstacles using the authors’ latency and success predictions.
Authors
Qi Wen, Borui Xu, Hussein Amro, Hongshi Tan, Hanmin Lin, Chaoyi Ruan, Shanghao Liu, Gang Yang, Yuejiang Liu, Jiafei Duan, Bingsheng He, Jason Clemons, Christos Kozyrakis, Wenqi Jiang
Abstract
Vision-language-action (VLA) models offer a promising path toward general-purpose robot manipulation. While inference latency can affect task success in a constantly evolving physical world, the relationship between latency and task success rate remains poorly understood. In this paper, we introduce VLALagBench, a comprehensive benchmark for studying VLA inference latency sensitivity across task types, object speeds, execution modes (synchronous and asynchronous), and VLA policies. Using VLALagBench, we find that even a modest inference latency of 100 ms can nearly halve task success on moving targets, with sensitivity varying substantially across tasks and policies. These differences reflect task-specific requirements for precision and continuous motion. By overlapping inference with action execution, asynchronous execution can recover some of the lost success at the cost of more frequent inference requests. Building on these findings, we develop an empirical model that predicts task success from limited calibration data, achieving root-mean-square prediction errors of only 5.7-7.8% across various VLA policies. Given an inference system's latency and per-robot throughput, these predictions guide the selection of system-specific VLA algorithm configurations to maximize task success. We believe VLALagBench and these findings provide a solid foundation for future robotics algorithm-system co-design.