Reward-robust rl improves learning by ignoring irrelevant reward changes
Behavior-Discriminative Reward Shaping for Reward-Robust Reinforcement Learning
Machine Learning
Summary
Reinforcement learning teaches a computer to make decisions by giving it rewards, but sometimes the rewards can be misleading or poorly designed. The authors found that many different reward systems can actually lead to the same behavior, so treating all reward differences as important makes the learning unnecessarily complicated. They created a method that groups rewards that lead to the same behavior and focuses only on truly different results, making learning more efficient and robust. Their approach works well in tests, helping computers learn better even when rewards are uncertain or noisy.
What this means in practice
- •For software engineers: Build reinforcement learning agents that remain effective despite errors in reward signals by using behavior-discriminative reward sets.
- •For robotics developers: Create robots that learn control actions reliably even when reward feedback is noisy or misspecified by applying shaping-aware robustness techniques.
Authors
Zixuan Liu, Fangzheng Wu, Brian Summa, Zizhan Zheng
Abstract
Reward-robust RL typically models misspecification by specifying an uncertainty set that constrains the discrepancy of plausible rewards from a base reward. However, such reward space discrepancies can be behaviorally irrelevant: under potential-based reward shaping (PBRS), many distinct rewards preserve policy ordering. This can introduce substantial redundancy into standard uncertainty sets and degrade optimization performance. We propose Shaping-Aware Reward-Robust RL, which constructs uncertainty sets over PBRS equivalence classes by projecting each reward to a canonical representative, ensuring that the resulting set contains only rewards that induce behaviorally distinct policy rankings. We prove that this projection preserves the optimal robust value while shrinking the uncertainty set and improving empirical performance. Using the connection between robustness and regularization, we obtain a practical algorithm to solve the shaping-aware reward-robust RL problem and enjoy convergence guarantees under standard assumptions. Experiments on benchmarks spanning diverse task domains and levels of complexity show consistent improvements over representative robust RL baselines and exhibit improved robustness to reward perturbations.