New way to learn what decisions a bandit algorithm believes are best
Learning from a Bandit Learner
Machine Learning
Summary
Bandit algorithms choose between options to get the best reward but often do not reveal how they view the quality differences between options. This paper focuses on learning what a bandit algorithm's internal beliefs about option quality are, rather than just the true rewards. The authors provide methods to estimate these beliefs from the choices the algorithm makes, especially when it randomizes choices like in Thompson sampling. They also explain the challenges and limits of this approach and test their methods on simulated examples.
What this means in practice
- •For machine learning engineers: Recover internal reward beliefs of bandit-based systems to better interpret and debug decision-making policies.
- •For online advertising teams: Infer the subjective value gaps that automated ad selection systems use when choosing which ads to show.
Tested on simulated data.
Authors
Michela Petriconi, Nicolas Menet, Claire Vernade
Abstract
Classical inverse bandit methods typically aim to recover true reward parameters. In contrast, we infer a demonstrator's subjective beliefs, specifically its internal estimates of suboptimality gaps. We consider no-regret demonstrators that follow randomized policies, such as Thompson sampling, and characterize a family of algorithms for which per-step inversion is possible. This enables gap estimates to be recovered directly from the arm-selection distribution. Our analysis identifies an inherent stability-coverage trade-off: inversion becomes ill-conditioned as the demonstrator concentrates its probability mass on a single arm, while the number of informative rounds depends on the rate at which its policy converges. We establish conditions under which inversion is reliable and provide theoretical guarantees on the resulting estimation error. Experiments on simulated data support the analysis and illustrate how these conditions affect gap recovery in practice.