Prioritized replay improves learning by correcting sample bias in stochastic outcomes
Correcting Within-Group Self-Selection Bias in Prioritized Replay
Machine Learning
Summary
In reinforcement learning, systems learn from past experiences by replaying important events. Sometimes, when outcomes are unpredictable, the way these experiences are chosen for replay can bias learning toward unusual results. The authors explain this bias and introduce methods that adjust replay to better reflect all outcomes fairly, improving learning efficiency. Their approach keeps focus on important experiences while correcting these biases to help algorithms learn more accurately.
What this means in practice
- •For machine learning engineers: Improve the training efficiency of reinforcement learning models in environments with unpredictable outcomes by using sibling-aware replay methods.
- •For game ai developers: Enhance game AI learning by correcting bias in replayed experiences, leading to more stable and accurate decision making under uncertain game dynamics.
Authors
Oscar Miró López-Feliu, Herke van Hoof
Abstract
Prioritized experience replay (PER) improves sample efficiency by replaying high-priority transitions, usually according to absolute temporal-difference error. In stochastic environments, PER can distort the distribution of realized outcomes replayed from transitions with the same state-action pair. We call this within-group self-selection. We quantify the resulting changes in within-group outcome frequencies and mean Bellman targets. We decompose PER into between-group allocation and conditional sibling selection, and derive fixed-buffer corrections that preserve current group-level priority mass: SAMPLE selects a group through PER and trains on a uniformly sampled sibling; AVG averages sibling Bellman targets; and MODEL samples from an empirical full-outcome model. In exact state-action environments with rare high-magnitude outcomes, sibling-aware replay improves learning efficiency over PER, although matched parameter sweeps show that tuning can narrow some gaps. In MinAtar, approximate VQ-VAE groups with SAMPLE mitigate degradation under mean-preserving reward tails in four of five games. Sibling-aware replay thus retains the focus on high-priority state-action regions while recovering their empirical outcome frequencies.