Explanations improve credit assignment in reinforcement learning for delayed rewards

Beyond Temporal Proximity: Explanatory World Models for Credit Assignment in Reinforcement Learning

Machine LearningArtificial IntelligenceInformation Retrieval

Summary

Reinforcement learning helps machines learn from rewards, but when those rewards come long after an action, it's hard to know which past actions mattered. The authors created a new approach called Explanatory World Models (EWM) that uses explanations to figure out which earlier actions likely caused later rewards. This method spreads the credit more accurately by using language models to generate these explanations. Testing shows EWM helps machines learn better in tasks where rewards appear after many steps, like recommendation systems and games.

What this means in practice

  • For recommender system engineers: Improve recommendation quality by better attributing user outcomes to earlier interactions using explanation-based credit assignment.$Commercial implications: Enables new generative recommendation products with improved accuracy by linking earlier user actions to delayed outcomes.
  • For game ai developers: Enhance reinforcement learning agents in long-horizon games by using explanations to assign credit for delayed rewards more effectively.

Authors

Oriol Corcoll, Qiyao Wei, Ciarán Mark Gilligan-Lee

Abstract

Generative Recommenders increasingly use Reinforcement Learning (RL) to optimize outcomes that may appear hundreds or thousands of interactions after a decision. Standard RL propagates reward according to temporal proximity rather than explicit relevance, attenuating the signal assigned to distant but important actions and creating a source of sample inefficiency. We introduce Explanatory World Models (EWM), an attribution framework that uses explanations to identify the earlier events that plausibly contributed to a realized outcome. We instantiate EWM with a frontier language model and use its explanations as learning signals using a conservative hierarchical operator that recursively redistributes reward while preserving the undiscounted episode return. To investigate what constitutes a useful explanation, we operationalize three complementary variants inspired by Halpern's actual causation, Lombrozo's explanatory simplicity, and Deutsch's hard-to-vary explanations. We evaluate EWM on two controlled long-horizon RL benchmarks, a public sequential-recommendation benchmark, and an industry-scale generative recommender. On Crafter and WordCraft, EWM substantially outperforms PPO and LaRe, with its largest gains when rewards are most delayed. On KuaiRand, EWM outperforms SFT and LaRe offline and A2C online. Finally, in an offline evaluation of an industry generative recommender, EWM considerably improves Recall and increases behavioral proxies. These results suggest that explanations can provide structured and reusable credit for learning from delayed outcomes.