Hybrid attention models speed training of diffusion language models
dQwen3.5: Hybrid-Attention Diffusion Language Models
Computation and LanguageMachine Learning
Summary
Training advanced language models that can generate text in flexible ways usually starts with a certain type of design called full-attention transformers. The authors looked at a different design that mixes attention with recurrent layers, which are harder to adapt but more efficient. They found that these hybrid models can be successfully turned into diffusion language models and learn faster, using fewer training examples to reach similar performance. This approach works across various model sizes, making it a promising direction to build efficient language models that can generate text in flexible orders.
What this means in practice
- •For machine learning engineers: Reduce training time and data needed to create diffusion language models by adapting hybrid attention and RNN architectures.
- •For natural language processing teams: Implement models capable of flexible text generation with faster training using hybrid backbones for diverse language tasks.
Authors
Anton Xue, Litu Rout, Aditya Akella, Adam Klivans, Sujay Sanghavi, Sanjay Shakkottai
Abstract
Adapting a pretrained autoregressive (AR) model is a cost-efficient route to a diffusion language model (DLM). While nearly all such adaptations start from a full-attention transformer, AR modeling has shifted toward hybrid architectures that interleave attention and RNN layers. This creates an obstacle for adaptation: unlike attention, RNNs are structurally causal and nontrivial to bidirectionalize. Despite this mismatch, we investigate whether such backbones can become effective DLMs by adapting Qwen3.5 at 0.8B, 2B, 4B, and 9B scales, yielding the dQwen3.5 family. We find that hybrid backbones can be efficient starting points for adaptation: against a full-attention control, the hybrid reaches a given training loss in about half the tokens. Across scales, dQwen3.5 resembles full-attention DLMs in any-order decoding behavior and performs strongly under parallel decoding.