Vision language action models latency depends more on strategy than size
VLA-HWBench: Where the Time Goes When VLA Models Run on Edge Hardware
RoboticsArtificial Intelligence
Summary
Running AI models that combine vision, language, and actions on small robot computers can be slow, which affects how well robots react in real time. The authors studied 18 such models on popular edge devices to see what affects speed the most. They found that how the model generates actions matters much more than how big the model is. Also, software tricks like making certain calculations run together and using smaller numbers help speed, but only for some models. Finally, the limits of the hardware, especially memory speed and CPU power, affect how fast these models run.
What this means in practice
- •For robot developers: Design VLA models and inference pipelines that generate longer action segments to achieve real-time control on edge devices.
- •For embedded system engineers: Extend kernel fusion and low-bit quantization techniques to speed up VLA model inference on various edge hardware platforms.
Authors
Borui Xu, Hongshi Tan, Wenqi Jiang, Bingsheng He, Ming Wu, Jason Zeng, Michael Heinrich
Abstract
Vision--language--action (VLA) models increasingly run on edge devices aboard robots, where inference latency constrains closed-loop robot control. However, it remains unclear how to design VLA models, their inference software and edge hardware so that robots can run VLA inference in real time. Most VLA models report only task success, while prior studies cover limited action-generation strategies, divide inference into at most three stages, and lack comparisons of inference optimizations on the same models.benchmarking 18 open VLA models across released and alternative backends on NVIDIA Jetson AGX Thor and AMD Ryzen AI Max+ PRO 395, with a seven-stage inference breakdown. Our study reveals three insights. First, action-generation strategy matters more than model size. It dominates 14 of 19 Jetson configurations, latency varies by $82\times$ on Jetson and $157\times$ on AMD, and parameter count explains at most 15% of latency variance. Models that generate long action chunks in a few flow-matching steps keep up with control, whereas token-by-token decoding does not. Second, kernel fusion with 4-bit weights helps most, running $π_{0.5}$ and GR00T N1.7 2.6--2.9$\times$ faster than PyTorch with \texttt{torch.compile}, but it exists for only these two models and only on the Jetson. Optimization also moves the time to the stages it shortens least, as CPU preprocessing grows from 12\% to 39\% of the GR00T N1.7 call. Third, after optimization, memory bandwidth limits only the stages that process few tokens on the Jetson, such as the denoising steps, whereas software limits all other stages, from unfused kernels to serial preprocessing on the host CPU. Overall, low-latency edge VLAs should use few action-generation iterations, inference software should extend kernel fusion and low-bit weights to more models and platforms, and edge hardware needs high memory bandwidth and a fast host CPU.