Instruction-Conditioned Exploration with Asymmetric Reinforcement Learning and Self-Distillation
2026-08-03 • Artificial Intelligence
Artificial IntelligenceComputation and LanguageMachine Learning
AI summaryⓘ
The authors explore how to make large language models learn better by encouraging them to try different ways of solving problems during training. They introduce a method called Instruction-Conditioned Exploration (ICE), which gives the model various instructions to increase the variety of behaviours it practices. Additionally, they create a new training technique called Asymmetric-RL/SD that helps the model remember and use these behaviours later when no extra instructions are given. Their approach improves the model's performance on challenging math reasoning tasks compared to previous methods.
Large Language ModelsReinforcement LearningExplorationInstruction-Conditioned ExplorationSelf-DistillationMathematical ReasoningPolicy TrainingQwen3-1.7BAction Space
Authors
Jim Dilkes, Vahid Yazdanpanah, Sebastian Stein
Abstract
Post-training Large Language Models (LLMs) with Reinforcement Learning (RL) has become an important tool for improving model capabilities, but the LLM action-space structure introduces challenges distinct from classical RL, with implications for inducing exploration. New methods are required that leverage the broad knowledge and flexibility of pre-trained LLMs to deliberately generate diverse experience at training time. We propose Instruction-Conditioned Exploration (ICE), which supplements task prompts during training with one of several distinct instructions, increasing the coverage of behaviours attempted. To facilitate ICE, we propose Asymmetric-RL/SD, a combined Reinforcement Learning and Self-Distillation training objective, to transfer explored behaviours to the unconditioned test-time policy. ICE with the Asymmetric-RL/SD objective improves Qwen3-1.7B held-out pass@1 performance at $4$K response length on mathematical reasoning tasks by $5.0\%$ relative to training with DAPO, with improvement persisting at a longer 8K context.