BlockGen: Flexible Blockwise Sequence Modeling with Hybrid Samplers
2026-06-01 • Machine Learning
Machine Learning
AI summaryⓘ
The authors studied two types of discrete diffusion models: uniform-state diffusion models (USDMs) and masked diffusion models (MDMs). They developed BlockGen, a new method that generates sequences block by block and allows mixing between different generation styles. BlockGen also introduces AR-informed predictor-corrector sampling (ARPC) to better fix likely mistakes during generation. Their experiments showed that USDMs generally perform better with simple sampling, but with ARPC and more steps, MDMs slightly outperform USDMs in some cases. This work clarifies when each diffusion paradigm works best in blockwise sequence generation.
uniform-state diffusion modelsmasked diffusion modelspredictor-corrector samplersblockwise sequence generationautoregressive modelsdiscrete diffusionperplexityancestral samplingGSM8K datasetGenerative Perplexity
Authors
Justin Deschenaux, Caglar Gulcehre
Abstract
Is the uniform-state diffusion framework a more powerful paradigm for discrete diffusion? Recent studies indicate that this may be the case. In combination with predictor-corrector samplers, uniform-state diffusion models (USDMs) produce samples of higher-quality than masked diffusion models (MDMs), and USDMs equal or outperform MDMs in downstream tasks, even though they exhibit greater perplexity. Two issues remain unresolved. First, existing work compares uniform and masked diffusion with un-informed correctors that re-inject noise at random positions, rather than targeting tokens most likely to be wrong. Second, prior work compares full-sequence diffusion models, so we do not know whether the same conclusion holds when tokens are generated block by block. To address these issues, we introduce BlockGen, a blockwise sequence model that we instantiate with both masked and uniform diffusion. BlockGen trains on a mixture of block sizes and its likelihood interpolates between AR and pure diffusion more finely than models with a fixed block size. BlockGen enables AR-informed predictor-corrector sampling (ARPC), which combines AR and diffusion predictions to re-generate unlikely tokens without an auxiliary verifier. Under ancestral sampling, uniform outperforms masked in the block-by-block setting, especially in the few-step regime. Under ARPC, the gap closes and reverses at high NFE. With block size $16$ on GSM8K, MDMs reach slightly higher accuracy than USDMs, and we observe a similar trend in Generative Perplexity on OpenWebText. Find our code at https://github.com/jdeschena/blockgen.