Stable multi-constraint reinforcement learning with orthogonalized updates
LOCU: Lowdin-Orthogonalized Constraint Updates for Multi-Constraint Policy Optimization
Artificial Intelligence
Summary
Sometimes, teaching a computer to make decisions while following many rules can be tricky because the rules interfere with each other. The authors found that using a special math trick called Lowdin orthogonalization helps untangle these interfering rules. This makes the computer’s learning process more stable and less likely to make mistakes when trying to satisfy many constraints at once. Their method, LOCU, also keeps things efficient by working in a smaller, simpler space. Tests show LOCU works well even when the rules are complex or hard to meet.
What this means in practice
- •For robotics engineers: Improve robot control systems that must obey multiple safety and performance constraints simultaneously using a numerically stable update method.
- •For autonomous vehicle developers: Enhance training of vehicle policies that must comply with diverse and interacting constraints like safety and energy efficiency using stable multi-constraint optimization.
Authors
Joonyoung Lim, Younghwan Yoo
Abstract
We investigate multi-constraint policy optimization in constrained Markov decision processes (CMDPs), where interactions among constraint gradients often yield ill-conditioned update directions, causing numerically fragile steps and unintended cancellation or redundant overlap in constraint corrections. To address this, we propose LOCU, which mitigates numerical instability arising from constraint interactions by applying symmetric Lowdin orthogonalization to the natural gradients of the reward and constraints under the Fisher-Rao metric. Combined with Fisher-geometric screening and near-collinear compression, LOCU yields a compact reduced space in which the trust region becomes a Euclidean ball and constraints are handled symmetrically through a low-dimensional active-set solve. For infeasible iterates, LOCU constructs a Pareto-descent direction which preserves the local feasibility of satisfied constraints within the same reduced formulation. Experiments show that LOCU remains stable under high constraint coupling, narrow feasible regions, and infeasible starts, and generalizes across different cost critic architectures.