Reinforcement learning improves exploration with anchored noise perturbations
Beyond Policy Rollouts: Embedding-Space Directional Exploration with Anchored Noise for LLM RLVR
Machine Learning
Summary
Reinforcement learning with verifiable rewards often struggles because it generates very similar attempts, making it hard to learn from differences. The authors propose a method called EDEN that slightly changes the input prompts with a bit of noise, helping the AI try different paths without changing how it normally works. This approach boosts the variety of solutions found and improves learning signals during training. These improvements hold across different model sizes and tasks involving mathematical reasoning.
What this means in practice
- •For machine learning engineers: Enhance reinforcement learning training by adding controlled input perturbations to prompts, increasing solution diversity and policy improvement signals.
- •For ai system integrators: Integrate exploration methods that perturb input embeddings to improve reasoning task performance without changing existing model architectures or inference procedures.
Authors
Daijiao Liu, Huiyi Wang, Lina Yao, Dong Gong
Abstract
Reinforcement learning with verifiable rewards (RLVR) typically uses small rollout groups per prompt, where repeated sampling from the same policy can yield highly similar trajectories, reducing within-group reward variation and weakening group-relative advantage signals. Existing remedies such as higher sampling temperature, entropy bonuses, asymmetric clipping, and larger rollout budgets broaden sampling or modify policy updates but leave rollout conditioning unchanged. In this paper, we propose EDEN (Embedding-space Directional Exploration via anchored Noise), which intervenes at the rollout source through anchored input perturbations. EDEN explores beyond the policy's own rollouts. Within the same rollout budget, it replaces a small fraction of standard rollouts in each group with rollouts that the same policy generates from the prompt plus a single appended, perturbed embedding position, rather than from the prompt alone. Gaussian noise drives variation, while a shared anchor provides a common conditioning direction within each prompt group. Selected perturbed rollouts contribute to GRPO updates with downweighted advantages, and the model architecture, discrete decoding, and test-time inference remain unchanged. Across model scales and mathematical reasoning benchmarks, EDEN outperforms GRPO and the exploration controls evaluated in the same pipeline, with the largest gains in sampled solution coverage. Analyses show that more prompt groups provide non-zero group-relative learning signals and that noise and anchoring play complementary roles. Intervening at the rollout source is thus an effective lever for exploration in RLVR.