Control method guarantees reaching and staying at goal states
A Universal Policy Wrapper With Guarantees
Machine Learning
Summary
Many tasks in robotics or automation require reaching a certain target and remaining there. The authors present a way to ensure that a controlling system reliably reaches and stays in this target area by combining a proven but imperfect fallback controller with a smarter but less reliable learned controller. Their method switches between these controllers based on how well the learned policy estimates success, offering performance benefits without sacrificing safety guarantees. Tests in simulations show that this approach balances good task performance with the certainty of reaching the desired goal.
What this means in practice
- •For robotics engineers: Deploy control systems in robots that combine learned behaviors with safety guarantees to ensure they reliably reach and maintain desired task goals.
- •For automation system designers: Create automated controllers for complex systems that achieve better performance while ensuring safe operation within required boundaries.
Authors
Georgiy Malaniya, Anton Bolychev, Pavel Osinenko
Abstract
Many control tasks require driving the system into a prescribed goal set and subsequently keeping it there. The considered setting assumes that a working controller is available that provably reaches and maintains the goal, albeit with suboptimal performance. This controller is referred to as the fallback policy. Conversely, although state-of-the-art reinforcement-learning (RL) algorithms can discover policies with impressive empirical performance, the neural networks at their core act as opaque black boxes learned through data-driven exploration and, as such, provide no theoretical mechanism to guarantee that the system will ever reach---and stay within---the prescribed goal set. This article presents CALF-Wrapper, a method that combines the fallback policy with a pretrained RL base policy and uses the critic estimate associated with the base policy to select between them during deployment. The theoretical analysis establishes that CALF-Wrapper inherits the goal-reaching property of the fallback policy under the stated assumptions and provides probability bounds for the settling time and the maximum distance from the goal set over a prescribed time interval. The experimental study evaluates CALF-Wrapper in four simulated control environments of different complexity. Comparisons include the base policy, fallback policy, and constrained reinforcement-learning methods formulated for the same objective. Sensitivity and ablation studies provide direct hyperparameter recommendations. The experiments show how CALF-Wrapper can provide effective task performance while attaining the maximum possible observed goal-reaching rate. CALF-Wrapper requires no additional dynamics model, online learning, or constrained optimization during execution.