Rethinking the Generation Order of Block Diffusion Language Models
2026-07-27 • Computation and Language
Computation and Language
AI summaryⓘ
The authors studied a new way to generate text using diffusion language models, which are a type of AI that can create text in flexible orders. They found that a newer type called block diffusion language models naturally work better when producing text from left to right, like normal reading. Using this insight, they created a new sampling method called Parallel Autoregressive Decoding (PARD) that is faster than usual but still produces high-quality text. Their experiments showed that PARD is better than other parallel methods and nearly as good as traditional left-to-right generation, but much faster.
Diffusion language modelsSampling methodsMasked diffusion modelsBlock diffusion language modelsParallel Autoregressive DecodingLeft-to-right decodingText generationAutoregressive decoding
Authors
Kai Syun Hou, James Kwok
Abstract
Diffusion language models enable flexible arbitrary-order generation, but existing sampling methods are mostly designed for early masked diffusion models (MDMs). In this work, we study sampling for recent block diffusion language models (BDLMs). We show empirically and analytically that these models are naturally more aligned with left-to-right decoding than MDMs. Based on this observation, we propose Parallel Autoregressive Decoding (PARD), a simple training-free sampling method that preserves left-to-right unmasking structure while allowing parallel token commitment. Extensive experiments show that PARD consistently outperforms existing parallel samplers in generation quality, while achieving substantial speedups over pure AR decoding with only a small quality gap.