Diffusion models speed up language generation and improve accuracy
How To Build A Modern Diffusion Language Model
Artificial Intelligence
Summary
Generating text one word at a time is slow and can't fix earlier mistakes. The authors describe diffusion models that create all words at once, then gradually improve them through several steps. This approach makes text generation much faster and allows correcting errors as it goes. These models have already grown very large and are being used in biology and language tasks.
What this means in practice
- •For natural language processing engineers: Build language generation systems that generate tokens in parallel, increasing inference speed and enabling error correction during output creation.
- •For bioinformatics tool developers: Create large-scale sequence modeling applications that utilize discrete diffusion to improve biological data generation and analysis.
A survey. It maps existing work.
Authors
Volodymyr Kuleshov, Marianne Arriola, Yair Schiff, Gilad Turok, Guanghan Wang
Abstract
The frontier of large language model (LLM) performance has advanced rapidly in recent years, fueled by scaling laws for pretraining on large text corpora. However, mainstream LLMs have fundamental limitations: they generate text autoregressively, producing outputs sequentially from left to right one token at a time. This sequential process leaves parallel GPU hardware underutilized during inference--a bottleneck that has grown acute as progress shifts toward scaling post-training and inference-time compute--and commits irreversibly to each token without the ability to revise earlier mistakes. Discrete diffusion models address these limitations by generating tokens in parallel, starting from a noisy or masked sequence and iteratively refining the entire output over a series of denoising steps. This parallel generative process enables speeds of thousands of tokens per second, provides a new axis of inference-time compute scaling, and supports built-in error correction and controllable generation. Over the past two years, discrete diffusion has scaled to billions of parameters across biological sequence modeling and natural language, with open-weights and commercial releases from leading labs. This article provides a survey of key ideas in discrete diffusion and a high-level introduction to how modern diffusion LLMs are built, covering their foundations, core algorithmic and architectural building blocks, applications in biology and language, and implications for future LLM scaling laws.