Non-Convex Sparse Reinforcement Learning via Non-Monotone Inclusions
2026-07-06 • Machine Learning
Machine Learning
AI summaryⓘ
The authors improve how a common method in reinforcement learning picks important features by using a special type of penalty that encourages sparsity but introduces some mathematical challenges. They handle these challenges by studying a broader category of problems that are not straightforward to solve. To solve these, they develop new mathematical rules ensuring their method converges properly. Their tests show this approach works better than current methods, especially when there is a lot of noisy data. Essentially, they connect new math theory with practical improvements in reinforcement learning.
Reinforcement LearningFeature SelectionLeast-Squares Temporal-Difference (LSTD)Non-Convex RegularizationProjected Minimax Concave PenaltyNon-Monotone InclusionsForward-Reflected-Backward Splitting (FRBS)Lyapunov StabilityMinty Variational Inequality
Authors
Kyohei Suzuki, onstantinos Slavakis
Abstract
This work delivers two key contributions: one to efficient feature selection in reinforcement learning (RL), the other to the theory of non-monotone inclusions. On the RL side, the estimation bias inherent in conventional regularization schemes is addressed by augmenting classical least-squares temporal-difference (LSTD) policy evaluation with the sparsity-inducing, non-convex projected minimax concave (PMC) penalty. Because the PMC penalty is weakly convex, the resulting fixed-point problem is no longer monotone; instead, it falls under a broader class of non-monotone inclusions involving the sum of a monotone Lipschitz operator and a hypomonotone operator. On the theory side, novel convergence conditions are developed for the forward-reflected-backward splitting (FRBS) method applied to this broader class of non-monotone inclusion problems. Under mild conditions, Lyapunov stability and the existence of a limit point of the sequence of FRBS iterates are established; alternatively, under the weak Minty variational inequality assumption, exact convergence is guaranteed. Numerical tests on benchmark datasets show that the proposed FRBS iterates, applied to the non-convexly regularized LSTD problem, substantially outperform state-of-the-art feature-selection methods, especially when many noisy features are present.