PriPG-RL: Privileged Planner-Guided Reinforcement Learning for Partially Observable Systems with Anytime-Feasible MPC
2026-04-09 • Machine Learning
Machine LearningRobotics
AI summaryⓘ
The authors study how to train robots or agents that can't see everything perfectly, by using a special helper during training that knows more about the environment. They create a method where a smart planning system with extra information guides the learning agent, which only has limited observations. Their approach, called Planner-to-Policy Soft Actor-Critic (P2P-SAC), helps the agent learn better and faster. They proved their method mathematically and tested it both in simulation and on a real robot walking through tricky obstacle courses.
Reinforcement LearningPartial ObservabilityPOMDPModel Predictive ControlSoft Actor-CriticPolicy DistillationSample EfficiencyRoboticsQuadruped RobotSimulation
Authors
Mohsen Amiri, Mohsen Amiri, Ali Beikmohammadi, Sindri Magnuśson, Mehdi Hosseinzadeh
Abstract
This paper addresses the problem of training a reinforcement learning (RL) policy under partial observability by exploiting a privileged, anytime-feasible planner agent available exclusively during training. We formalize this as a Partially Observable Markov Decision Process (POMDP) in which a planner agent with access to an approximate dynamical model and privileged state information guides a learning agent that observes only a lossy projection of the true state. To realize this framework, we introduce an anytime-feasible Model Predictive Control (MPC) algorithm that serves as the planner agent. For the learning agent, we propose Planner-to-Policy Soft Actor-Critic (P2P-SAC), a method that distills the planner agent's privileged knowledge to mitigate partial observability and thereby improve both sample efficiency and final policy performance. We support this framework with rigorous theoretical analysis. Finally, we validate our approach in simulation using NVIDIA Isaac Lab and successfully deploy it on a real-world Unitree Go2 quadruped navigating complex, obstacle-rich environments.