AI summaryⓘ
The authors study a method called speculative decoding, which speeds up language model output by letting a simpler model (drafter) suggest multiple words at once for a bigger model to check quickly. They found that sometimes the drafter suggests whole blocks too easily, missing out on possible speed gains, which they call 'stranded speed-up.' To fix this, the authors developed a retraining method called DBloom that helps the drafter handle longer word blocks better. This approach improved the number of tokens accepted early during generation across different models and benchmarks, outperforming some prior methods. Their work shows how carefully expanding and retraining drafters can enhance efficiency in language generation.
speculative decodingdrafter modelblock diffusioncommitted lengthacceptance histogramtoken generationpost-trainingcurriculum learninglanguage modelsQwen and Gemma models
Abstract
Speculative decoding speeds up generation with an efficient draft model (drafter) that proposes tokens for a target model to verify in one pass, preserving the target's output distribution. High-acceptance block-diffusion drafters such as DFlash and DFlare fill an entire block in one parallel pass. In many cycles, the target accepts the whole block, so the drafter exhausts its trained block horizon before verification fails. We call this unrealized acceptance stranded speed-up. A mean committed length, per prompt or per cycle, hides it, whereas the acceptance histogram exposes it as a spike in the ceiling bin, the fraction of cycles that accept the entire block. We recommend the histogram as a preflight check before spending training compute. Naively widening the block at inference does not recover the speed-up, because once the block outgrows its training size, the drafter's bidirectional attention shifts its distribution even at early positions and erodes front-of-block verification. Instead, we post-train the drafter on a longer block with a short curriculum that emphasizes the newly exposed positions, a method we call DBloom. Expanding the pretrained DFlash and DFlare drafters from block size 16 to 24 across Qwen3-8B and Qwen3-4B targets raises the per-prompt committed length on the high-ceiling benchmarks by a median of +0.8 tokens (up to +1.1). Once continuation fine-tuning precedes expansion, the increase reaches 1.37 tokens. The same expansion also lifts committed length on all seven benchmarks for Gemma-4-12B-IT, a different model family, by a median of +0.41 tokens (Arm A), and the full continuation-then-expand pipeline (Arm B) adds +0.29 to +0.98 tokens over the same B16 drafter. In a prompt-matched comparison against JetSpec, a contemporary tree-based drafter not used in our design, DBloom commits more tokens on every benchmark at tree budgets up to 64 nodes.