Large language models speed up text generation with block proposals
ProVe: Proposer-Verifier Decoding
Computation and LanguageMachine Learning
Summary
Generating text using large language models can be slow because words are predicted one by one. This paper introduces a method called ProVe, where the model predicts a group of words all at once, then quickly checks if they make sense. This speeds up the process without losing accuracy or with a small, controllable drop in performance. The approach can balance faster text production against slight differences in output quality.
What this means in practice
- •For ai model engineers: Increase language model inference speed by designing models that generate multiple tokens simultaneously, improving throughput without extra draft training.
- •For cloud service providers: Offer faster text generation APIs by integrating a proposer-verifier decoding scheme that balances speed and accuracy for large models.$Commercial implications: Enables commercialization of faster language model inference services with adjustable quality, appealing to customers needing scalable NLP solutions.
Authors
Ziqi Jin, Weiwen Xu, Jiaxi Li, Yujie Zhong, Qi She, Aixin Sun
Abstract
Speculative decoding accelerates autoregressive (AR) language models by letting a small model draft tokens serially and a large model verify them in parallel. We reverse these roles. Starting from a standard AR checkpoint, we fine-tune the large model into a proposer that predicts a block of future tokens in a single forward pass, then use one additional forward pass to verify the proposed block. This gives ProVe, Proposer-Verifier decoding. The verifier can be a smaller AR model or the proposer itself; the latter gives ProVe-lossless, which reproduces the proposer's greedy AR decoding. Block proposals already provide substantial room for acceleration: with a block size of 16, the proposed block has an average AR-matching prefix of 6.6 tokens on GSM8K. Across six benchmarks with a 32B proposer, ProVe-lossless reaches 2.5x average AR throughput without training a separate draft head. With a small 3B verifier, ProVe reaches 2.7x average throughput at 2.6 accuracy points below AR on the six-task average, providing a controllable tradeoff between speed and accuracy.