Vision language models shape robot actions under timing limits
Beyond LLM Serving: Characterizing Vision-Language-Action Workloads for Embodied AI System Design
Hardware ArchitectureRobotics
Summary
Robots need to quickly understand what they see and say to act in real time. The authors studied how vision-language-action models perform on different hardware when robots must make decisions under strict time limits. They found that how big the action data is changes whether memory or computing speed matters most and showed that energy use and speed must be balanced carefully. The study helps design smarter robots by matching the model, hardware, and timing strategies.
What this means in practice
- •For robotics system designers: Optimize hardware and model choices to meet robot action timing without staleness for better task success.
- •For embedded device engineers: Adjust GPU frequency and resource allocation to balance energy use and latency in low-latency robot inference.
Authors
Seonghun Jung, Sieun Moon, Jiyoung Jeong, Jimin Lee, Jaehyuk Huh
Abstract
Vision-language-action (VLA) models translate multimodal observations into low-level robot actions. During robot operation, each control period sets an inference deadline, and overruns leave the robot acting on stale observations, reducing task success. Meeting this deadline motivates on-device or nearby edge execution, where a single robot requires batch-1 inference outside the design point of LLM serving systems. Although VLA architectures combine familiar vision-language, autoregressive, and diffusion-style components, their runtime behavior in this batch-1 control setting remains uncharacterized. We characterize four representative VLA models on an edge GPU server and two onboard SoCs, using single-inference profiling and 43,200 closed-loop episodes. Action tensor dimensionality determines whether a stage is memory- or compute-bound, platform balance can shift that bottleneck, and GPU frequency scaling yields a platform-dependent energy-latency sweet spot. In closed-loop operation, overlapping inference with action execution creates an accuracy-speed-energy tradeoff, and no configuration is Pareto-dominant across deployment SLOs. These results guide joint design of VLA model architectures, hardware, and runtime policies.