Representation guides faster and better parallel text generation
Representation-based Masked Diffusion Model
Computation and Language
Summary
Generating sentences where many words change at once can confuse computers because those words depend on each other. The authors found a way to help computers understand the overall meaning of the sentence first and then update the words together more smoothly. They do this by turning the sentence into a math-friendly code that keeps track of the big picture. With this, their method creates clearer and more accurate sentences, especially when changing many words quickly.
What this means in practice
- •For natural language processing engineers: Use representation-guided masked diffusion to generate text faster and with better quality when updating multiple words simultaneously.
- •For machine translation developers: Improve the coherence of translations generated in parallel by conditioning on global semantic information during token updates.
Authors
Yangrong Hu, Ding Huang, Xueyu Zhou, Jian Huang
Abstract
Masked Diffusion Models (MDMs) have emerged as a compelling paradigm for language modeling, offering the capability for efficient parallel text generation. However, existing parallel sampling methods typically update multiple masked tokens independently and ignore the complex mutual dependencies among the masked tokens. This independent updating mechanism lacks global coordination and might lead to incoherent outputs. To address this limitation, we propose Representation-based Masked Diffusion Model (RMDM), a framework that leverages the text representation to explicitly encode global semantics and help to parallel update tokens more precisely. Specifically, we first encode text into a continuous semantic space using a pretrained encoder and learn an invertible transformation that normalizes the representation distribution to a Gaussian prior, facilitating efficient sampling during generation. Conditioned on this latent semantic representation, we train a masked diffusion model to learn the conditional text distribution, where the representation serves as global semantic guidance to coordinate parallel token updates and faithfully approximate the target distribution. Empirical results demonstrate that RMDM significantly improves generation quality, particularly in aggressive few-step sampling regimes.