Counterfactual Shapley Credit Assignment
Machine LearningArtificial Intelligence
Summary
The authors address a key problem in reinforcement learning: figuring out which actions truly lead to success versus what happens due to chance. They propose a new method called Counterfactual Shapley Credit Assignment that uses ideas from causal theory to fairly assign 'credit' to an agent's decisions, separating skill from luck. Their approach improves credit assignment especially when rewards are rare, outcomes are random, or rewards come after many actions. They also create an efficient way to calculate this credit and show that it helps learning agents perform better in difficult tasks.
Authors
Mingxuan Li, Kaizhan-Lee, Elias Bareinboim
Abstract
The Credit Assignment Problem (CAP) is fundamental to developing efficient and explainable Reinforcement Learning (RL) agents. Existing frameworks, whether relying on temporal contiguity or hindsight-conditioned reward reweighting, frequently fail to attribute properly between an agent's policy (skill) and environmental stochasticity (luck). A principled approach to CAP must isolate the true causal drivers of observed outcomes from spurious correlations and environmental randomness. We introduce Counterfactual Shapley Credit Assignment, a novel framework grounded in causal theory that attributes credit and blame via the Counterfactual Shapley Value ($φ$-value). By redistributing environmental rewards, $φ$-values enhance temporal credit assignment across three critical dimensions: sparse causality, high stochasticity, and delayed rewards, all while preserving the optimal policy. We derive a consistent estimator that computes $φ$-values efficiently, enabling a new class of policy gradient methods, $φ$-PPO, combined with Prioritized Trajectory Replay (PTR). Empirical results demonstrate that $φ$-values align precisely to the ground truth causes of task rewards with superior sample efficiency in challenging environments where prior state-of-the-art methods fail to converge.