Papers for

ai services engineers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

DScale improves large language model decoding speed by nearly half

DScale: Scaling Block-Diffusion Speculative Decoding with Adaptive Verification

Abstract: Growing large language model applications demand efficient inference. At high concurrency, block-diffusion speculative decoding suffers from verification padding, rejected candidates, and incompatibility between variable prefixes and fixed-shape graphs. Uniform truncation sacrifices acceptable tokens. We present DScale, preserving drafter architecture, weights, and full draft length. A separate 112K-parameter predictor requires neither confidence calibration nor hardware speed-curve preparation. Path-aware tiles reduce padding. Dynamic verify-length (DVL) allocation packs scored prefixes into half the native verification capacity. Fixed-address workspaces propagate changing boundaries through verification and acceptance while reusing captured graphs. On A100-40GB with tensor parallelism 1, Qwen3-8B and Qwen3-4B cover four datasets and concurrency 8-32, reusing each target's frozen predictor. Geometric-mean throughput gains across these configurations are respectively 43.9% and 48.8% over DFlash, 22.2% and 37.7% over DSpark, and 24.4% and 32.0% over Domino, with lower request latency. Cumulative ablations show that adding the three mechanisms successively increases geometric-mean throughput, while budget adjustment improves accepted-token retention. GPU profiling shows that complete decode-step time on GSM8K decreases by 30.8-52.5% relative to DFlash

Tue 29 SeptDistributed, Parallel, and Cluster ComputingArtificial Intelligence
The gist
Large language models need to generate words quickly, especially when many requests happen at once. The authors show that existing methods for speeding this up can waste time or miss good guesses. They created DScale, a technique that keeps all the original guessing steps but uses smarter checks and better packing of information to reduce wasted work. This results in faster responses with less delay on powerful GPUs.
Open → 2609.37532v1