Faster vision language model deployments trade speed for task accuracy
When Faster VLA Deployment Changes Closed-Loop Behavior: Task Success-Latency Analysis of SmolVLA Across PyTorch and ONNX Variants
RoboticsMachine Learning
Summary
The paper looks at how speeding up a vision-language-action (VLA) model affects its performance on tasks. The authors compare two ways of running the model—PyTorch with automatic mixed precision (AMP) and ONNX Runtime with different precision settings—on a simulated robot task. They find that certain fast ONNX settings reduce the time needed but also lower success in spatial tasks, though not object tasks. Adjusting how much language context the model uses helps regain some accuracy, showing context size is an important factor in performance.
What this means in practice
- •For robotics developers: Optimize vision-language models' deployment settings to balance task speed and accuracy in simulated robot control environments.
- •For machine learning engineers: Improve task success in latency-sensitive applications by tuning language context width when deploying vision-language models with ONNX.
Authors
Rafiqul Islam
Abstract
Vision-language-action (VLA) deployment can reduce inference latency while changing closed-loop task behavior. We evaluate HuggingFaceVLA/smolvla_libero on an RTX 2060 (6 GB) in LIBERO Spatial and Object (MuJoCo 3.3.2, LeRobot 0.6.1, seed 42), comparing PyTorch+AMP with ONNX Runtime CUDA Execution Provider (CUDA EP). The main evaluation uses 100 episodes/suite; a paired rollout uses 300 episodes/suite. PyTorch+AMP reaches 70.0%/88.0% Spatial/Object success at 1181 ms p99. Requested-FP16 and requested-INT8 ONNX reduce tether-inspect p99 to 601 ms and 532 ms, while Spatial success falls to 41.0% and 40.0% and Object remains at 89.0%. A graph audit shows those artifacts are byte-identical FP32 graphs, so the requested-INT8 row is not operator-level INT8 quantization. A static language-width ablation (16/24/32 tokens) yields Spatial success of 41.0%, 75.0%, and 71.0%; widths 24 and 32 recover much of the Spatial drop while Object success and uniform-bench latency stay approximately stable. Width-24 ONNX Spatial success is comparable to the PyTorch+AMP baseline at roughly half the latency (Wilson intervals overlap; two-proportion chi-squared p=0.53). Context width is an important contributor in this stack; it does not account for every PyTorch-vs-ONNX difference. Deployment evaluation should jointly report latency, artifact inspection, interface constraints, and closed-loop success. Code: https://github.com/rafiqul713/smolvla-libero-onnx.