Jev decision model improves reinforcement learning efficiency and exploration
Can Jev be Your Q or Policy in Reinforcement Learning?
Machine LearningArtificial Intelligence
Summary
Reinforcement learning (RL) systems often learn slowly and struggle to adapt to new tasks. The authors study a new kind of decision model called Jev that can make decisions all at once rather than step-by-step. They find that using Jev in different roles within RL training helps the system learn faster and explore better without needing to train the Jev model itself. This approach works well on various simulated tasks and games, showing promise for improving RL training.
What this means in practice
- •For game ai developers: Use Jev as a pretrained decision model in training RL agents to achieve faster learning and better exploration in game environments.
- •For robotics software teams: Integrate Jev as a fixed decision component to improve sample efficiency and learning performance in robotic control tasks.
Authors
Yi Ma, Tianpei Yang, Yaodong Yang, Weixun Wang, Hongyao Tang
Abstract
Foundation models supply reinforcement learning (RL) with priors that mitigate its longstanding weaknesses in sample efficiency and transfer, but their token-by-token generation makes queries sequential and costly. Jev, a recently released decision model, generates nothing and returns calibrated, typed answers in a single forward pass. Existing work studies foundation models in RL either as models to be trained or as generators to be prompted, and Jev belongs to neither category, having so far served only as a black box in single domains. How well such a model decides on its own in RL environments, and how it can improve RL as a component of training, therefore remain unaddressed. To this end, in this paper we first examine the requirements that the objects of an RL system place on the answers they consume, and establish that Jev can fulfill all of them except the cardinal use of a value function. The remaining objects form positions that admit several roles each. We then construct algorithms that employ Jev at three of these positions, as a reference policy, an exploration judge, and a replay rater, to improve sample efficiency, exploration, and learning performance. Across nine MiniGrid tasks and three Atari games, training with Jev outperforms a standard RL learner, including where the learner makes no progress alone, while the model itself remains untrained. To our knowledge, we present the first use of Jev within the RL learning process and establish a frozen decision model as a usable component of RL training, inviting further exploration of how Jev and other advanced decision models can improve RL.