Evaluating covariate balance for long time horizon Markov decision processes
Machine Learning
Summary
The authors looked at how well certain tests, called covariate balance diagnostics, work to find hidden problems in studies that use offline reinforcement learning (RL) to suggest treatments. They found that either many of these studies might have big biases, or the tests themselves might not be good enough to check for problems. Because of this, we can't say these studies are very reliable yet. The authors suggest new ways to improve how offline RL is used for treatment recommendations.
Authors
Joshua Spear, Rebecca Pope, Neil J Sebire
Abstract
This article explores the application of covariate balance diagnostics for detecting the presence of hidden confounding/model miss-specification in studies applying offline reinforcement learning (RL) to deriving optimal treatment recommendations. The results demonstrate that, either there is a high risk of bias within existing offline RL studies for treatment recommendations or, existing covariate balance metrics are not sufficient to assess such studies. Regardless, existing offline RL studies cannot be concluded as being statistically robust. The conclusions propose future research directions for obtaining more methodologically robust applications of offline RL to treatment recommendation problems.