Adaptive decoding boosts speed by predicting multiple tokens at once
Next Token or Vector? Learning How Far to Decode
Machine LearningArtificial IntelligenceEmerging Technologies
Summary
Standard AI text generation predicts one word at a time, which can be slow even when several upcoming words could be guessed together. The authors propose a method called CADENCE that learns how many words to predict at once based on the context. Their model uses special vectors to represent groups of future words and a router to decide the best prediction length. This approach speeds up text generation and improves prediction accuracy by verifying each guess carefully.
What this means in practice
- •For ai software engineers: Implement faster language model decoding by adaptively predicting multiple tokens at once to increase generation throughput without losing accuracy.
- •For mobile app developers: Speed up on-device text generation in language-based apps by using context-adaptive token prediction to reduce latency.
Authors
Sai Sreenivas Chintha, Chidera Biringa, Rishab Sharma, Anusha Agrawal, Prachi Badarayani, Gemma Zhang, Nigel Boachie Kumankumah, Marco Ayala, Sulaiman Vesal
Abstract
Standard autoregressive decoding advances one token at a time, even when several upcoming tokens can be predicted together. We propose CADENCE (Context-Adaptive DEcoding), a generation method that learns both what to draft and how far to draft. Gaussian prediction heads represent future spans as continuous latent vectors, and a shared decoder converts the selected vector into token predictions in parallel. A learned router selects the prediction length from the model's own hidden states and predicted Gaussian parameters, adapting generation granularity to the current context with single-token generation as a fallback. The backbone verifies each proposed continuation and retains the prefix that matches its greedy predictions. Experiments with Llama-3.1-8B-Instruct on Alpaca, GSM8K, and HumanEval show that router-selected positions yield higher draft acceptance than applying the corresponding prediction length everywhere. CADENCE achieves up to 1.3x higher generation throughput than standard greedy decoding.