Primal dual learning shows resonance effects in changing constrained systems
Parametric Resonance in Primal-Dual Learning for Slowly Non-Stationary Constrained MDPs
Machine Learning
Summary
When making decisions that have strict rules that can change slowly over time, a common technique called primal-dual learning can behave unpredictably. The authors studied how changes in rule levels or sensitivities can cause the learning steps to resonate, or amplify, leading to instability. They explain that some oscillations are cancelled out naturally, but instability arises if the two learning parts don’t use matching information. They also demonstrated these effects in a reinforcement learning agent and showed how different adjustments can suppress or maintain these resonances.
What this means in practice
- •For reinforcement learning engineers: Improve stability of constrained RL algorithms when environment constraints vary slowly by tuning updates to avoid parametric resonance.
- •For control system designers: Design controllers managing constraints with slowly changing parameters to anticipate resonance instability from dual-primal mismatches.
Authors
Maxime Baelde
Abstract
Primal-dual methods for constrained Markov decision processes are analyzed under a two-timescale assumption, with non-stationarity entering through two channels. In the additive channel, where the constraint level oscillates, the dual oscillation saturates at a floor set by the moving optimum and is amplified linearly by the timescale ratio. In the parametric channel, where the constraint sensitivity oscillates, a coherent modulation carries no resonant forcing at any depth: the dual update's stiffness modulation cancels the primal update's damping modulation. Instability instead comes from a mismatch between the constraint estimate used by the primal update and the measurement used by the dual update, producing a parametric resonance at twice the pair's natural frequency. The tolerated mismatch grows with the square root of the forcing bandwidth and collapses with the square root of the dual-to-primal step-size ratio. These results hold on the tangent linear-quadratic reduction around an active saddle, for one constraint and for $K$ constraints under a common modulation and an isotropy condition. Driving both updates from the same estimate removes the resonant forcing but introduces a stationary offset equal to the estimator's bias. In a Lagrangian soft actor-critic agent, the multiplier locks onto the sub-harmonic at the timescale ratios its identified linear loop predicts and a coherent cost critic suppresses the lock-in, while a proportional (PID) term, predicted to leave the lock-in in place, suppresses most of it through the projection of the multiplier.