Curriculum reinforcement learning improves reasoning in diffusion language models
CanvasAnneal: Curriculum Reinforcement Learning for Diffusion Language Models
Machine Learning
Summary
Diffusion Language Models can generate text in parallel but struggle with complex reasoning tasks compared to traditional autoregressive models. The authors propose CanvasAnneal, a method that helps these models learn reasoning better by starting with hints from a stronger teacher model and gradually having the model work more independently. This approach speeds up learning and improves performance on tasks like math problems and tool usage. Their work suggests guiding training with structured examples can help overcome exploration difficulties in reinforcement learning for language models.
What this means in practice
- •For ai developers: Improve training efficiency and reasoning ability of diffusion-based language models for complex tasks such as math problem solving and tool interaction.
- •For software engineers: Develop language systems capable of faster learning and better logical reasoning using structured curriculum guidance during reinforcement learning.
Authors
Blake Olson, Yuhang Song, Emmett McQuinn, Yuan Shangguan
Abstract
Diffusion Language Models (DLMs) offer promising parallel generation capabilities but lag behind autoregressive models in complex reasoning and tool-use tasks. While Reinforcement Learning (RL) has recently been applied to enhance DLMs, standard RL approaches suffer from an exploration bottleneck. To address this, we inject reasoning priors from a stronger teacher model to guide RL exploration. In this paper, we introduce CanvasAnneal, a curriculum-guided diffusion RL framework. During the initial RL phase, we warm-start exploration by injecting teacher-generated reasoning traces into the initial diffusion canvas. As training progresses, we gradually remove this guidance and require the model to generate more of the reasoning trajectory independently. Across mathematical reasoning and tool-use benchmarks, CanvasAnneal improves over standard diffu-GRPO on MATH500, Countdown, and Tau2 and substantially accelerates reward improvement on several tasks, while gains are task-dependent. Our results suggest that structured training-time guidance can alleviate exploration bottlenecks in diffusion RL and speed up convergence on harder tasks.