Zarya language model blends sequential and parallel text generation
Zarya: A Hybrid Autoregressive--Masked Diffusion Language Model with Flexible Training and Dual-Mode Inference
Computation and LanguageArtificial Intelligence
Summary
Generating text sequentially, one word at a time, is slow but consistent, while parallel methods can be faster but often less coherent and costly to run. The authors created Zarya, a new language model that combines both approaches, training it to gradually shift from careful step-by-step learning to broader, parallel learning. At use time, Zarya can generate text either by sampling in parallel or by mixing parallel and sequential steps efficiently. This flexible system aims to balance speed, memory use, and output quality in language tasks.
What this means in practice
- •For natural language processing engineers: Implement text generation systems that exploit both sequential and parallel decoding modes to trade off speed and coherence dynamically.
- •For machine learning infrastructure teams: Optimize large-scale deployment by reusing cache in mixed decoding modes while maintaining flexible model configurations for various workloads.
Authors
Leonid Sinev, Ilya Koziev, Vladislav Leshchuk
Abstract
Autoregressive language models (ARMs) are constrained by sequential, left-to-right generation, while masked diffusion models (MDMs) enable parallel decoding but suffer from high computational overhead due to the inability to reuse Key-Value (KV) cache and from incoherent generation arising from learning dependencies over an intractable space of token combinations. We introduce Zarya, a family of hybrid language models that jointly optimizes an autoregressive (AR) objective and a masked-diffusion objective within a single architecture. Zarya structures training data into variable-size slots and employs a curriculum that gradually increases slot granularity, enabling a smooth transition from fine-grained AR learning to coarse-grained diffusion learning. At inference, Zarya provides two distinct decoding paradigms through a unified interface: (i) MDM sampling with first-hitting denoising, and (ii) slotted speculative decoding that interleaves inter-slot diffusion-based selection with intra-slot autoregressive infilling, achieving full KV cache reuse. The training and inference regimes are fully decoupled, allowing a model trained with any configuration to be deployed in either mode. Extensive configurability --- including grouped noise patterns (Prefix Completion, Fill-In-the-Prefix, Fill-In-the-Middle), ordered sampling schedules, and noise-level permutation strategies --- enables flexible research exploration. We release Zarya models publicly in sizes 0.6B, 1.7B, and 4B, demonstrating performance on standard benchmarks while offering a principled integration of autoregressive and diffusion paradigms.