Speculative decoding speeds up audio understanding on mobile devices
AS$^2$D: Accelerating On-Demand Audio Understanding on Mobile Devices
SoundArtificial Intelligence
Summary
Decoding speech into text on phones can be slow because each step depends on the previous one. The authors found that when understanding audio, the prediction step doesn’t always need to wait for every past word to be confirmed. They created AS2D, a method that lets the prediction and verification steps work at the same time, making processing faster without losing accuracy. Testing on several phones and datasets showed it greatly improves speed for tasks like speech recognition.
What this means in practice
- •For mobile app developers: Improve real-time speech-to-text features on smartphones by using asynchronous token prediction to speed up decoding without hardware upgrades.
- •For voice assistant engineers: Enhance voice assistant responsiveness by enabling parallel processing of audio input decoding and verification on-device.
Authors
Yunzhe Li, Kyoungjun Park, Hongzi Zhu, Lili Qiu
Abstract
Speculative decoding accelerates autoregressive generation by using a smaller drafter to propose tokens for batched verification by a larger target. However, conventional speculative decoding couples drafting to the target's evolving verified prefix, serializing drafting and verification. We ask whether this dependency is necessary for source-conditioned generation. Our key observation is that, for audio language models, the input audio and user request can provide useful speculative candidates without following the target's evolving text prefix. We propose AS$^2$D (Audio Speculative Speculative Decoding), which enables target-decoupled drafting: an audio-conditioned drafter follows its own generation history while the target independently verifies and corrects ready candidates. Without usable candidates, the target advances alone. Thus, target feedback determines which candidates are committed but no longer determines when the drafter can make progress, enabling drafting and verification to proceed concurrently while retaining target-side verification and correction. We implement AS$^2$D in MNN for Android and evaluate two target models across four phones, seven datasets, and three tasks covering 12.2 hours of audio. Across four phones, AS$^2$D improves pooled ASR throughput by 42-76% over target-only decoding, while only 5.7% of evaluation windows are slower than target-only, compared with 58.1-63.0% for speculative baselines. For ASR, AS$^2$D reaches 97.33-98.20% of a hindsight per-window oracle's pooled throughput over the evaluated drafter/budget catalog. Native on-demand execution with a 7B target achieves up to 78% higher throughput than target-only. These results show that source-conditioned audio generation can relax the conventional dependence of speculative drafting on the target's evolving output prefix, exposing substantial parallelism for efficient inference.