Minimal intervention improves reinforcement learning exploration effectively
MInTRL: Off-policy Intervention can boost On-policy RL
Machine LearningArtificial Intelligence
Summary
Reinforcement learning often faces a choice between exploring new actions and sticking to known ones. The authors introduce a method called MInTRL that gently corrects mistakes during learning rather than letting the system fail fully or relying too much on outside knowledge. This approach helps the learning system explore more possibilities while still focusing on its own behavior. Their tests show it works better than common methods on tasks involving math and coding.
What this means in practice
- •For autonomous system developers: Improve robot or agent training by adding minimal interventions to help them explore better without large disruptions.
- •For software developers for ai assistants: Enhance coding or math assistant AI performance by integrating minimal corrections during training to broaden their problem-solving abilities.
Authors
Mingyu Chen, Yefan Tao, Gerald Friedland, Xuezhou Zhang, Chris Kong
Abstract
Reinforcement learning with verifiable rewards is typically performed on-policy, keeping training data close to the current policy but limiting learning to trajectories that the policy can discover itself. Off-policy methods such as supervised fine-tuning, on the other hand, can leverage external knowledge beyond the base model's capabilities, but may suffer from large distribution shift. The key challenge is thus to expand exploration without sacrificing learnability. In this work, we introduce Minimal Intervention Reinforcement Learning (MInTRL), which expands the exploration frontier through sparse, local interventions in otherwise on-policy rollouts. During generation, a judge-intervention policy periodically reviews the current policy's output, replaces erroneous suffixes with short corrections, and immediately returns control to the policy. During training, MInTRL adopts a sequence-level advantage-regression objective that eliminates the need for importance sampling. We show that sparse, local interventions can substantially improve coverage beyond finite-budget on-policy sampling while preserving the overall on-policy nature of the resulting trajectories. Across math and code benchmarks, MInTRL consistently outperforms standard on-policy and off-policy baselines. Ablations show that MInTRL remains effective with self-intervention and across different judge policies, while performance peaks at moderate intervention intensity, highlighting the importance of intervening minimally. These results establish minimal intervention as an effective paradigm for enhancing on-policy RL.