Continuous flow language models improve reasoning with new reward method
Backpropagating Verifier Rewards Through Continuous Flow Language Models
Machine Learning
Summary
Language models can generate text one word at a time, but some newer models create smooth word-like signals instead of discrete words, making it tricky to train them with rewards. The authors propose a method that helps these smooth models learn from feedback by estimating values for partial word signals before turning them into actual words. This method lowers training noise and improves the models' reasoning abilities on several tasks compared to other similar approaches.
What this means in practice
- •For natural language processing engineers: Improve language model reasoning by fine-tuning continuous embedding models with low-variance reward signals.
- •For ai chatbot developers: Enhance chatbot responses by incorporating reinforcement learning techniques for continuous embedding language models.
Authors
Xu Han, Jiajing Hu, Li-Ping Liu
Abstract
Reinforcement learning (RL) has substantially improved the reasoning of autoregressive and discrete-diffusion language models. Language models that generate continuous token embeddings are emerging as a strong alternative, but RL post-training for them remains understudied. We propose \method{}, a post-training framework for continuous flow language models. Such a model denoises token embeddings over multiple steps and decodes them into discrete tokens only at the last step. To obtain pathwise gradients, which backpropagate through continuous actions and have much lower variance than REINFORCE, we introduce a critic that estimates the value of intermediate token embeddings, bypassing the non-differentiable decoding step. The critic must also keep track of the value function as the policy changes; to this end, we design a critic that predicts a value from every token representation and improves policy optimization. Across four reasoning tasks, \method{} reliably improves a continuous flow language model through RL post-training and outperforms baseline methods of the same type.