Robotic muscle memory speeds up vision language model inference
rMuscle: Robotic Muscle Memory for Efficient Vision-Language-Action Model Inference
RoboticsArtificial Intelligence
Summary
Robots that do repetitive factory tasks rely on vision-language models to understand what they see and decide what to do next. Running these models takes time, which can slow down the robots and make their movements jerky. The authors found that many robot tasks are very similar each time they run, not just in what they see and do, but also inside the model's internal workings. They created rMuscle, a system that remembers these similarities like muscle memory, reusing parts of past computations to speed up the robot’s thinking without making mistakes.
What this means in practice
- •For robotics engineers: Enable robots in factories to perform repeated tasks faster by reducing the time their AI models take to process visual and action data.
- •For embedded device developers: Improve real-time AI inference speed on devices like Jetson Thor by reusing model outputs and activations during repetitive actions.
Authors
Kaijun Zhou, Zhiyang Li, Le Chen, Jinyu Gu
Abstract
Factory work is a promising early scenario for embodied AI: assigning repetitive manual jobs to robots has clear economic payoff, and a structured station keeps the jobs tractable for current policies. Vision-Language-Action (VLA) models now dominate as the policy paradigm for these robots. The inference latency of VLA models directly affects robot responsiveness and motion smoothness. However, existing VLA inference frameworks do not fully exploit the characteristics of embodied workloads or account for the distinct bottlenecks across different stages of VLA inference. In this paper, we first characterize embodied workloads and identify substantial task similarity across repeated robot executions. We further find that such similarity extends beyond observations and action trajectories to internal model states. Drawing on these observations, we present rMuscle, a real-time VLA inference framework inspired by human muscle memory. It exploits cross-execution similarity through a dual-phase muscle-memory cache. The Context Cache reuses visual-token outputs to reduce computation, while the Action Cache reuses neuron activation patterns to reduce weight accesses. We keep both the cache memory footprint and access overhead low through online cache recomputation, sliding-window cache retrieval, and mask sharing across consecutive denoising steps. rMuscle achieves 1.29-1.42X speedup on RTX 4090 and Jetson Thor across LIBERO, RoboTwin, and physical manipulation tasks, while maintaining the original success rates on real-world robots.