Diffusion large language model decoding speeds up with tentative context pipelining
TeCoP: Accelerating Diffusion Large Language Model Decoding via Tentative Context Pipelining
Computation and Language
Summary
Generating text with special large AI models called diffusion large language models can be slow because they can’t fully use their predictions until they are certain about them. The authors created a method called Tentative Context Pipelining (TeCoP) that lets the model use partially confirmed parts of the text earlier to speed up the process. This approach cuts down the number of repeated calculations needed and boosts the overall speed without losing accuracy. Tests show significant improvements across various models and benchmarks.
What this means in practice
- •For ai system engineers: speed up inference pipelines by reducing redundant computations in diffusion language model decoders using tentative context shared across blocks
- •For cloud ai service providers: improve throughput of hosted large language models by integrating TeCoP decoding to serve more requests per time unit$Commercial implications: Enables faster text generation services that can be marketed as higher performance and lower cost AI offerings.
Authors
Jui-Chien Tsou, Hsuan-Chi Liu, Wei-Chin Wang, Yu-Syuan Xu, Chun-Yi Lee
Abstract
Diffusion large language models (dLLMs) predict multiple token positions per forward pass, yet block-wise decoders serialize generation and repeatedly recompute stable predictions. We attribute this inefficiency to coupling a prediction's readiness for irreversible commitment with its utility as context. We introduce Tentative Context Pipelining (TeCoP), a training-free decoder that decouples these roles through staggered block-parallel pipelining and draft-and-verify. TeCoP reuses stable but uncommitted predictions as provisional cross-block context, with commitment governed by confidence and stability checks. Across four LLaDA and Dream dLLMs on GSM8K, Minerva MATH, and MBPP, TeCoP reduces number of forward evaluations by up to 87% and achieves up to 7.50-fold throughput without KV caching and 11.77-fold with caching over the uncached original decoder. It outperforms Fast-dLLM and DSB in throughput across all 12 model-benchmark settings, with and without caching, while keeping average cached accuracy within one percentage point of the original decoder.