Care-vi improves reliability in off-policy actor-critic learning
CARE-VI: Conservative Adaptive Reliability Estimation for Value Improvement in Off-Policy Actor-Critic Learning
Machine Learning
Summary
Improving decision-making in AI agents relies on accurately estimating values of future actions, but noisy data can lead to premature or biased choices. The authors created CARE-VI, a framework that carefully ranks and reviews possible actions, adjusts corrections based on reliability, and avoids overconfidence in value estimates. This approach helps AI agents learn better policies by making more trustworthy predictions of future rewards. Experiments on standard robotic tasks demonstrated that CARE-VI outperforms existing methods by producing higher average returns.
What this means in practice
- •For reinforcement learning engineers: Enhance the stability and performance of off-policy actor-critic algorithms in robot control tasks by improving value target reliability.
- •For autonomous system developers: Use the CARE-VI framework to improve decision-making reliability in systems requiring adaptive control based on uncertain data.
Authors
Xiang Zou, Shengzhu Shi, Junqi Gao, Zhichang Guo
Abstract
Reliable temporal-difference targets are central to off-policy actor-critic learning. Direct value improvement refines the next-state target with alternative actions, but the reliability of this refinement depends on how candidate actions are ranked, reviewed, and weighted. Noisy rankings may force premature candidate commitment, reusing selection scores may bias target valuation, and fixed enhancement weights may amplify weak evidence. To address these risks, we develop Conservative Adaptive Ranking and Screening (CARS), which retains an ordered candidate prefix within a preset budget and narrows it only when the observed boundary gap exceeds a disagreement-scaled uncertainty radius. Selector-Evaluator Value Assessment (SEVA) uses selector critics to order candidates and a separately parameterized evaluator critic to review the selected value, then caps the reviewed value at the selector reference. Dynamic Adaptive Risk-aware Enhancement (DARE) then regulates each residual correction using candidate reliability, the gap between selector and evaluator signals, and a finite stage factor. Together, CARS, SEVA, and DARE form CARE-VI, an evidence-regulated target construction framework that preserves the backbone interfaces for critic regression and actor updates. The analysis bounds the CARS boundary error, the SEVA selected-value overestimation, and the one-sided deviation of the DARE residual displacement from its population counterpart, and establishes fixed-policy recovery after the finite-stage perturbation ends. Experiments with SAC, TD3, and TD7 on four MuJoCo tasks show that CARE-VI achieves the highest mean return in all twelve settings. Grouped ablations and scalar diagnostics support the roles of the three components in improving target reliability.