DScale improves large language model decoding speed by nearly half
DScale: Scaling Block-Diffusion Speculative Decoding with Adaptive Verification
Distributed, Parallel, and Cluster ComputingArtificial Intelligence
Summary
Large language models need to generate words quickly, especially when many requests happen at once. The authors show that existing methods for speeding this up can waste time or miss good guesses. They created DScale, a technique that keeps all the original guessing steps but uses smarter checks and better packing of information to reduce wasted work. This results in faster responses with less delay on powerful GPUs.
What this means in practice
- •For ai services engineers: Reduce latency and improve throughput for concurrent large language model requests using block-diffusion speculative decoding.
- •For high-performance computing teams: Use adaptive verification techniques to maximize GPU efficiency when running large-scale language model inference pipelines.
Authors
Rongjian Chen, Minxian Xu, Zhengxin Fang, Kejiang Ye, Chengzhong Xu
Abstract
Growing large language model applications demand efficient inference. At high concurrency, block-diffusion speculative decoding suffers from verification padding, rejected candidates, and incompatibility between variable prefixes and fixed-shape graphs. Uniform truncation sacrifices acceptable tokens. We present DScale, preserving drafter architecture, weights, and full draft length. A separate 112K-parameter predictor requires neither confidence calibration nor hardware speed-curve preparation. Path-aware tiles reduce padding. Dynamic verify-length (DVL) allocation packs scored prefixes into half the native verification capacity. Fixed-address workspaces propagate changing boundaries through verification and acceptance while reusing captured graphs. On A100-40GB with tensor parallelism 1, Qwen3-8B and Qwen3-4B cover four datasets and concurrency 8-32, reusing each target's frozen predictor. Geometric-mean throughput gains across these configurations are respectively 43.9% and 48.8% over DFlash, 22.2% and 37.7% over DSpark, and 24.4% and 32.0% over Domino, with lower request latency. Cumulative ablations show that adding the three mechanisms successively increases geometric-mean throughput, while budget adjustment improves accepted-token retention. GPU profiling shows that complete decode-step time on GSM8K decreases by 30.8-52.5% relative to DFlash