SpecPilot boosts large language model decoding speed over twofold
SpecPilot: Autonomous Algorithm Discovery and Systems Optimization for Speculative Decoding
Machine Learning
Summary
Decoding language models can be slow because each step needs a lot of memory access, limiting speed. The authors found that existing fast methods don't fully realize their potential due to verification and batching issues. They created SpecPilot, a smart agent that automatically discovers better algorithms and system tweaks for each specific setup. By checking outputs carefully and learning from past successes, SpecPilot more than doubles the speed advantage of speculative decoding without lowering answer quality.
What this means in practice
- •For machine learning engineers: Optimize deployment of language models to increase decoding speed while maintaining output quality through automated system tuning.
- •For cloud infrastructure teams: Enhance GPU utilization and reduce latency for language AI services by applying adaptive speculative decoding optimizations.
Authors
Kunjun Li, Wenhao Chai, Yihao Liang, Yuezhou Hu, Shuyao Xu, Yi Pan, William X. Zheng, Kannan Ramchandran, Kurt Keutzer, Jiantao Jiao
Abstract
Speculative decoding accelerates memory-bound LLM decoding with a lightweight drafter, and parallel drafters extend this speedup by drafting a whole block in one forward pass. However, it remains unclear whether their released systems (stock) attain the full speedup that parallel drafting permits. We identify three challenges. First, with one request, each step accepts few tokens but reads every target weight. Second, with batching, per-request verification work roughly halves the speedup over autoregressive (AR) decoding. Third, speculative decoding is deployment-specific, and no universal optimization fits every serving regime, drafter, target model and GPU. To address these challenges, we present SpecPilot, an autonomous agentic loop that discovers algorithms and systems optimizations for each deployment. In each round, a cost model locates the largest gaps between the current system and its ceiling. Parallel agents explore directions and integrate the successful ones. Token forcing checks every output token against stock to detect reward hacking. A structured memory carries validated mechanisms to new rounds and deployments. SpecPilot discovered a general lossless 12-bit weight format, tree verification for hybrid recurrent targets, two-level verification and three batching mechanisms. We evaluate it across four target models, three drafters, two GPU platforms and five benchmarks. Without training or quality loss, it raises the speedup over AR from stock's 4.46$\times$ to 10.87$\times$ with one request, and from 1.94$\times$ to 3.93$\times$ with batching.