Speech conditioned robot models reduce delay in following commands
Late Breaking Results: Audio-Language-Action Models with Direct Speech Conditioning for Robotic Manipulation
Computer Vision and Pattern Recognition
Summary
Robots often need to understand spoken instructions, but usually this requires turning speech into text first, which can slow things down and cause mistakes. The authors created a new system called Speech-π0.5 that lets robots convert raw speech directly into movement commands, skipping the need for text. This makes robots respond faster while still performing tasks reliably, even understanding accents they haven't heard before. They tested their system on a robot manipulation benchmark and found it cuts response time by three times compared to traditional methods.
What this means in practice
- •For industrial automation teams: Enable faster, more accurate robot control from spoken instructions in manufacturing settings without text delay.
- •For smart home device manufacturers: Integrate direct speech-to-action models to improve real-time responsiveness of voice-controlled home robots.$Commercial implications: This paper enables products that handle speech commands faster and more robustly, enhancing marketable home robot responsiveness.
Authors
Enfu Nan, Pu Zhao, Yixiao Chen, Lin Zhao, Juyi Lin, Chen Wang, Weiwei Chen, Yanzhi Wang
Abstract
Vision-language-action (VLA) models enable robots to follow instructions in typed text, limiting broader deployment in natural speech. The traditional solution by prepending a speech recognition before VLA to convert speech into texts would incur additional latency and propagate errors. To address this, we present Speech-$π_{0.5}$, which maps raw speech directly to motor commands without explicit textual transcriptions. Specifically, built on $π_{0.5}$, we use a cross-modal projection to convert voice frames directly into audio tokens with significant response latency reduction. To further improve the accuracy, we adopt split LoRA adaptation with dedicated audio and task adapters, under a two-stage training. Speech-$π_{0.5}$ achieves $3\times$ lower response latency than ASR pipeline while maintaining competitive task success on LIBERO, and generalizes robustly to unseen accents where a single adapter degrades significantly.