Always-on testing keeps improving smart treatments safely over time
Always-On Experimentation
Artificial Intelligence
Summary
As computers rapidly create and test new ideas like medicines or ad campaigns, experiments now run all the time, adding and dropping treatments as they work or not. The paper studies how to decide if each new treatment is really helpful without making too many mistakes. The authors build new math tests that carefully track the results as new treatments come and go, so decisions remain reliable no matter when they stop or change the tests. Their approach uses a special betting idea to keep the error rate under control even as experiments run forever.
What this means in practice
- •For online marketing teams: Confidently add and remove advertising strategies while experiments run continuously without increasing false positives.
- •For pharmaceutical data scientists: Test new drug candidates one by one within an ongoing trial while controlling error rates to avoid false efficacy claims.
Authors
Ricardo J. Sandoval, David Arbour, Avi Feller, Michael I. Jordan
Abstract
Generative AI has dramatically accelerated the rate at which new treatments---from novel pharmaceuticals to online marketing campaigns---can be conceived and deployed. As a result, modern experimentation platforms often run continuously, with treatments added as they are ready and removed when they underperform. We formalize this "Always-On" experimental setting, in which treatments can be dynamically generated, added to, and removed from a running experiment, and study the statistical problem of deciding whether to accept or reject each treatment while controlling for the false discovery rate. We develop sequential tests that achieve time-uniform Type-I error control under arbitrary stopping times and "predictable" treatment schedules. Our approach builds on the testing-by-betting framework: we construct test supermartingales for testing the average treatment effect of each treatment, and show that the construction of these test supermartingales is growth-rate optimal in an almost-sure sense.