Efficient long text generation using compressed diffusion blocks
Denoising Blocks, Not Tokens: Efficient Compressed Continuous Diffusion with Branching Token Realization
Machine LearningArtificial Intelligence
Summary
Generating long text quickly is challenging because typical methods handle each word one by one. The authors introduce a new approach that groups multiple words into blocks and processes these blocks together, greatly speeding up text generation. They use a technique called Branching Latent Diffusion to compress long sequences into fewer parts, then decode each part independently and in parallel. This method significantly improves speed and reduces computing effort while keeping the text quality reasonably good for local details, though maintaining overall story coherence remains hard.
What this means in practice
- •For natural language processing engineers: Boost text generation speed for long documents by compressing sequences into blocks and decoding them in parallel.
- •For machine translation teams: Generate translated text faster by applying block-level diffusion to reduce computation time while maintaining local fluency.
Authors
Xinsong Feng, Peng Du, Zhizhuo Yang, Daniel M. Bikel, Jiayun Wang, Haipeng Chen
Abstract
Diffusion language models (DLMs) generate text through iterative parallel refinement, offering the potential for higher throughput than autoregressive (AR) decoding. However, most DLMs still maintain one generative state per token, so every denoising step processes a state sequence as long as the output sequence, limiting the throughput gains from parallel generation. Continuous DLMs provide an additional degree of freedom: a single continuous state can represent multiple tokens, allowing diffusion to operate on a much shorter latent sequence. We introduce \emph{Branching Latent Diffusion (BLD)}, which exploits this flexibility by compressing a 1024-token sequence into only 64 block latents, a $16\times$ reduction. BLD combines latent compression with \emph{branching token realization}, where each latent is decoded by a local AR branch and all branches run in parallel. Because strong compression makes joint latent generation difficult, BLD generates the latents in groups, conditioning each group on previously generated latents. In end-to-end evaluation on the same GPU, BLD reduces generation FLOPs by more than $80\times$ and increases throughput by more than $6\times$ relative to the similarly sized ELF-L baseline. Compared with the AR baseline, BLD achieves more than $6\times$ higher throughput and more than $4\times$ lower latency. Despite the compression, BLD maintains competitive local fluency and diversity, although long-range coherence remains challenging. Overall, BLD shows that moving diffusion from token-level states to compressed latent sequences can substantially improve the efficiency of long-sequence generation.