Model speeds up large language output by filtering low-quality guess branches
ValuSpec: Plug-and-Play Candidate Valuation before Target Verification for Tree-Based Speculative Decoding
Artificial Intelligence
Summary
Large language models try out many possible next words in parallel to speed up generating text, but some of these guesses are low quality and waste time checking. The authors propose ValuSpec, a method to quickly estimate which guesses are worth keeping before doing the slower, detailed checks. ValuSpec learns from the model’s own behavior without extra training and ensures the final output matches what the model would normally produce. This approach speeds up text generation in several benchmark tests by over twice as fast.
What this means in practice
- •For ai service operators: Accelerate inference of large language models by filtering low-quality candidate outputs before verification to reduce computation costs.
- •For cloud computing providers: Improve efficiency of language model serving infrastructure by integrating a plug-and-play candidate filter that cuts unnecessary verification work.
Authors
Liang He, SiYuan Ma, Qishi Zhan, Yongqi Fan, Mingyu Cao, Qizhen Lan, Zhaolu Kang, Xilu Wang
Abstract
Tree-based speculative decoding accelerates large language model inference by verifying multiple candidate branches in parallel. However, in practical serving scenarios, similar requests repeatedly induce overlapping candidate branches. While retrieval-based methods mitigate this by reusing historical drafts, the resulting candidate trees often contain many low-quality branches as the candidate tree grows, which reduces verification efficiency and wastes target-side computation. To address this, we propose ValuSpec, a plug-and-play candidate valuation mechanism that filters low-quality branches from a supplied candidate tree before expensive target-side verification. ValuSpec introduces transfer-based supervision to align filtering with the target model's behavior. A filtering threshold is estimated from the target model's greedy trajectories on a calibration split, without any parameter training. The target model then verifies the filtered tree under the output-identical greedy rule, so accepted tokens are guaranteed to match standard autoregressive decoding exactly. Experimental results on HumanEval, GSM8K, Dolly, and SpecBench show that, in our main evaluation, ValuSpec achieves up to 2.38x end-to-end speedup in retrieval-based tree settings. Furthermore, integrating ValuSpec into the EAGLE-3 dynamic tree framework yields up to 2.78x end-to-end speedup without modifying its tree construction strategy, demonstrating that ValuSpec serves as a generally applicable and plug-and-play candidate valuation module.