Robots improve actions by switching vision models based on failure prediction

VLA Handover: Failure-Prediction-Based Switching from On-Robot to Server VLA Models

RoboticsMachine LearningNetworking and Internet Architecture

Summary

Robots can run small vision-language-action (VLA) models onboard quickly but sometimes fail to act correctly. Larger models on remote servers work better but are slower and costly when used for every action. The authors propose having the robot use the small model most of the time and switch to the big model only when failure is likely. This approach raises success rates and reduces how often the robot needs to ask the server, improving speed and efficiency.

What this means in practice

  • For robotics engineers: Use failure predictions to switch between onboard and server VLAs, improving robot action success without constant server queries.
  • For network operations teams: Reduce peak server load by controlling when robots send requests for large VLA inference remotely based on predicted failures.

Authors

Takayuki Nishio

Abstract

Large vision-language-action (VLA) models generalize well but are difficult to run in real time on a robot, and small VLAs that run there fail more often. Remote inference over beyond-5G and 6G networks lets a robot use a large VLA on a server, but querying the server for every action chunk limits the number of robots per server. We propose VLA handover: the robot runs a small VLA (S-VLA) and switches to a large VLA (L-VLA) on a server only when the S-VLA is likely to fail. We formulate the switching as minimizing the request cost under a required success probability and, since this problem cannot be solved directly, propose a heuristic framework that retrains neither VLA. In simulation with SmolVLA and $π_{0.5}$ on three of the four LIBERO suites, VLA handover raised the success rate by 11-22 percentage points over the S-VLA alone and sent 49-77% fewer requests than server-only control at a 2-4 percentage-point lower success rate, and its completion time grew less with the network latency.