Q-based Variational Inverse Reinforcement Learning
2026-08-17 • Machine Learning
Machine Learning
AI summaryⓘ
The authors present a new method called QVIRL for teaching AI systems what humans want by learning from examples, without needing explicit instructions. Their approach learns not just a single reward idea but a range of possible rewards, helping the AI understand uncertainty better. Unlike past methods, QVIRL works well with complex tasks, including video game environments and raw pixel data. This makes it useful for situations where safety and adaptability are important.
Inverse Reinforcement LearningBayesian MethodsVariational InferenceQ-valuesReward FunctionsApprenticeship LearningActive LearningGridworldLunar LanderATARI Games
Authors
Ondrej Bajgar, Peter Tisnikar, Alessandro Abate, Konstantinos Gatsis, Maike Osborne
Abstract
The development of safe and beneficial AI requires that systems can learn and act in accordance with human preferences. However, explicitly specifying these preferences by hand is often infeasible. Inverse reinforcement learning (IRL) addresses this challenge by inferring preferences, represented as reward functions, from expert behaviour. We introduce Q-based Variational IRL (QVIRL), a novel Bayesian IRL method that recovers a posterior distribution over rewards from expert demonstrations via primarily learning a variational distribution over optimal Q-values. Unlike previous approaches, QVIRL combines scalability with uncertainty quantification, important for safety-critical applications as well as active learning. We demonstrate QVIRL's strong performance in apprenticeship learning across various tasks, including gridworlds, Lunar Lander, the Highway Environment, and two ATARI games both with static expert data and with active learning. It is the first method for Bayesian IRL that demonstrates training from raw pixel observations.