Papers for
product testing teams
Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.
Principal agent testing expands to ongoing experiments with adaptive stopping
Principal-agent hypothesis testing under adaptive experimentation
Abstract: Principal-agent hypothesis testing studies statistical contracts in which a principal rewards an agent according to data the agent generates. Existing work focuses on the single-shot setting, where the agent produces a single dataset. We extend the framework to sequential settings, where evidence accumulates over time, rewards may be deferred to the end or disbursed incrementally, and the agent may stop the contract at a strategically chosen stopping time. We call a contract null-deterring if no ''null agent'' (those whose data-generating distribution lies in the null hypothesis) can profit in expectation under any stopping rule, and our main conceptual result shows that such contracts are characterized by e-processes. We then turn to the agent's best response and identify the optimal stopping rule through the Snell envelope of the reward process. We develop simpler, more interpretable approximations that often require the agent to have only partial knowledge of its distribution.
Adaptive sampling method cuts experiments to confirm good designs faster
Halve-Then-Greedy: Adaptive Sampling for Posterior Success with Near-Optimal Instance Dependence
Abstract: We study fixed-budget adaptive experimentation aimed at maximizing the probability of obtaining sufficient posterior evidence that a recommended design's mean performance exceeds a target. Unlike best-arm identification, this objective does not require finding the best design. For Gaussian bandits with a Gaussian prior, we propose \emph{Halve-Then-Greedy} (HTG), which combines a large initial balanced allocation, successive halving, and posterior-greedy sampling. We prove instance-dependent, finite-budget exponential bounds on posterior-failure probability that capture both separation from unfavorable arms and the presence of multiple favorable arms. On one-positive-arm families, we establish an asymptotic sample-complexity lower bound that HTG matches up to a logarithmic factor. We further analyze uniform sampling and sequential halving, revealing regimes where they require more samples than HTG by linear and logarithmic factors, respectively. Simulations illustrate HTG's sample-efficiency advantages and their dependence on instance structure.