Diffusion model generates entire text sequences in one step
One-Step Text Generation by Seq-Drifting
Artificial Intelligence
Summary
Creating text by computers often takes many steps to sound natural and accurate. The authors developed a new approach called Seq-Drifting, which teaches a model to produce a whole sentence or paragraph all at once. It uses a system that guides the choice of each word based on the ones around it, avoiding repeating the same words too much. Their method works well on different writing tasks like translation and summarization.
What this means in practice
- •For natural language processing teams: Produce entire text outputs instantly rather than through iterative steps for faster language model deployment.
- •For machine translation engineers: Generate full-length translated texts in a single pass, improving speed in translation systems.
Authors
Jiayu Liu, Ruizhi Zhao, Tianyun Ji, Yee Whye Teh, Zhenya Huang, Ning Miao
Abstract
Diffusion language models generate tokens in parallel, but still require multiple sampling steps to produce high-quality text. In this paper, we investigate whether a model can learn to generate an entire sequence in a single forward pass. We propose Seq-Drifting, a framework that applies drifting to sequences of continuous token representations and directly learns a map from Gaussian noise to text. Unlike existing DLMs that derive the supervision at each position directly from the observed token in a training sequence, Seq-Drifting builds a drift field whose position-wise targets adapt to the current generated context. At each position, attraction guides each output toward a set of contextually plausible token representations, while repulsion discourages collapse across samples and repetition within sequences. The generator learns from this drift field during training and produces all tokens jointly in one pass at inference. Empirically, Seq-Drifting achieves a generative perplexity of 36.01 on LM1B in one step and performs competitively in both unconditional and conditional generation. Experiments on translation, summarization, and reasoning further demonstrate the applicability of the framework beyond open-ended generation. These results suggest that drifting over token sequences offers a promising path toward effective one-step text generation. Our code is available at https://github.com/jyliu-ai/Seq-Drifting.