Diffusion language models improve token prediction with context adaptive coupling
Learning to Couple: Context-Adaptive Joint Denoising for Diffusion Language Models
Machine Learning
Summary
Language models that generate text sometimes guess multiple words at once but treat those guesses as unrelated, which can cause errors. The authors propose Codpula, a new method that lets the model consider how these guesses connect based on the surrounding text. This approach adjusts predictions dynamically and keeps the math consistent, improving accuracy without slowing down the process. Codpula performs better on tasks requiring reasoning, math, and coding compared to previous models.
What this means in practice
- •For ai developers: Build faster language models that generate multiple related tokens accurately in parallel, enhancing capabilities for reasoning and code generation.
- •For software engineers: Improve code autocompletion tools by integrating context-adaptive joint denoising to reduce errors when predicting multiple code tokens simultaneously.
Authors
Zilei Shao, Ian Li, Heng Zhao, Kewei Lian, Guy Van den Broeck, Zhe Zeng, Anji Liu
Abstract
Diffusion language models accelerate generation by predicting multiple tokens in parallel, but factorized denoisers treat simultaneous predictions as conditionally independent. Decoding heuristics reduce the resulting errors by controlling which tokens are committed, but leave the underlying distribution factorized. Joint modeling addresses the source of the problem, yet existing methods attach separately learned layers to frozen denoisers, leaving the backbone unadapted to the joint objective and the coupling parameters fixed across contexts. We propose Codpula, a context-parameterized denoiser combining a Transformer with a tractable probabilistic inference layer. Given a partially masked input, the Transformer produces token potentials that shape per-position predictions and compact gates that modulate the layer's coupling parameters, yielding a context-dependent joint distribution while retaining exact conditional inference. We jointly optimize both components under a conditional denoising objective using a dual-flow multiplicative update that preserves valid probabilistic parameters. At inference, Codpula supports standard heuristics and uses conditional queries to adapt commitments to dependencies among proposed tokens without extra Transformer evaluations within a step. Codpula sets new state-of-the-art results across a broad suite of reasoning benchmarks while advancing the accuracy-latency Pareto frontier in mathematical reasoning and code generation.