Self-speculative decoding speeds up large language model text generation
SEED: Self-Speculative Decoding via Implicit Encoder-Decoder
Computation and LanguageMachine Learning
Summary
Generating text with large language models can be slow because they predict one word at a time using deep computations. The authors present a way called SEED that reuses intermediate results already computed to draft multiple words quickly before checking for accuracy. This approach balances speed and quality better than previous methods and works by treating parts of the model like an encoder and others like a decoder in a clever way. Their experiments show SEED makes text generation nearly three times faster on some models without losing quality.
What this means in practice
- •For ai platform engineers: Speed up language model inference by quickly drafting multiple tokens using cached deep representations, reducing computational resources in cloud deployments.
- •For chatbot developers: Improve response speed by generating text faster without sacrificing quality, enabling more fluid and responsive conversational agents.
Authors
Hankun Lin, Patrick Pynadath, Ruqi Zhang
Abstract
Self-speculative decoding accelerates large language model (LLM) inference by drafting tokens from the target model itself, but faces a sharp tradeoff between the quality and cost of the draft. Early-exit methods produce drafts cheaply by terminating computation at intermediate layers, but forgo the deeper representations that later layers provide and thus suffer in draft quality. Multi-token prediction preserves draft quality by emitting from the model's final hidden states, but pays for a full forward pass to produce those states at every drafting step. We propose self-speculative encoder-decoder (SEED), a self-speculative method that obtains high-quality drafts cheaply by reusing the deep contextual representations already computed during verification. We reinterpret the standard decoder-only transformer as an implicit encoder-decoder: the first layers (encoder) build deep contextual representations, and the last few layers (decoder) emit tokens from them. Encoding and verification are merged into a single step: verification is performed by the full encoder-decoder, and the contextual representations of the verified prefix are cached for reuse during drafting. Drafting is therefore very fast: between verifications, the lightweight decoder drafts multiple tokens autoregressively, each conditioned on the cached representations and on preceding drafts. Experiments across multiple benchmarks show that SEED achieves up to 2.7$\times$ average speedup on 4B-scale models, outperforming both early-exit and MTP-style self-speculative baselines and running 28% faster than the state-of-the-art EAGLE-3, while preserving or even improving the generation quality of standard autoregressive fine-tuning. Code is available at https://github.com/lhk2004/SEED.